How to open a large text file on Linux

In the era of "big data", large text files (GB or more) could be commonly encountered around us. Suppose you somehow need to search and edit one of those big text files by hand. Or you could be analyzing multi-GB log files manually for specific troubleshooting purposes. A typical text editor may not be designed to deal with such large text files efficiently, and may simply get choked while attempting to open a big file, due to insufficient memory.

If you are a savvy system admin, you can probably open or touch an arbitrary text file with a combination of cat, tail, grep, sed, awk, etc. In this tutorial, I will discuss more user-friendly ways to open (and possibly edit) a large text file on Linux.

Vim with LargeFile Plugin

Vim text editor boasts of various plugins (or scripts) which can extend Vim's functionality. One such Vim plugin is LargeFile plugin.

The LargeFile plugin allows you to load and edit large files more quickly, by turning off several Vim features such as events, undo, syntax highlighting, etc.

To install the LargeFile plugin on Vim, first make sure that you have Vim installed.

On Debian, Ubuntu or Linux Mint:

$ sudo apt-get install vim

On Fedora, CentOS or RHEL:

$ sudo yum install vim-enhanced

Then download the LargFile plugin from Vim website. The latest version of the plugin is 5, and it will be saved in Vimball format (.vba extension).

To install the plugin in your home directory, you can open the .vba file with Vim as follows.

$ gunzip LargeFile.vba.gz
$ vim LargeFile.vba

Enter ":so %" and press ENTER within Vim window to install the plugin in your home directory.

After this, enter ":q" to quit Vim.

The plugin will be installed at ~/.vim/plugin/LargeFile.vim. Now you can start using Vim as usual.

What this plugin does is to turn off events, undo, syntax highlighting, etc. when a "large" file is loaded on Vim. By default, files bigger than 100MB are considered "large" by the plugin. To change this setting, you can edit ~/.vimrc file (create one if it does not exist).

To change the minimum size of large files to 10MB, add the following entry to ~/.vimrc.

let g:LargeFile=10

While the LargeFile plugin can help you speed up file loading, Vim itself still cannot handle editing an extremely large file very well, because it tries to load the entire file in memory. For example, when a 1GB file is loaded on Vim, it takes as much memory and swap space, as shown in the top output below.

So if your files are significantly bigger than the physical memory of your Linux system, you can consider other options, as explained below.

glogg Log Explorer

If all you need is "read-only" access to a text file, and you don't have to edit it, you can consider glogg, which is a GUI-based standalone log analyzer. The glogg analyzer supports filtered views of an input text file, based on extended regular expressions and wildcards.

To install glogg on Debian (Wheezy and higher), Ubuntu or Linux Mint:

$ sudo apt-get install glogg

To install glogg on Fedora (17 or higher):

$ sudo yum install glogg

To open a text file with glogg:

$ glogg test.log

glogg can open a large text file pretty fast. It took me around 12 seconds to open a 1GB log file.

You can enter a regular expression in the "Text" field, and press "Search" button. It supports case-insensitive search and auto-refresh features. After searching, you will see a filtered view at the bottom window.

Compared to Vim, glogg is much more lightweight after a file is loaded. It was using only 83MB of physical memory after loading a 1GB log file.

JOE Text Editor

JOE is a light-weight terminal based text editor released under GPL. JOE is one of few text editors with large file support, allows opening and editing files larger than memory.

Besides, JOE supports various powerful text editing features, such as non-destructive editing, search and replace with regular expression, unlimited undo/redo, syntax highlighting, etc.

To install JOE on Debian, Ubuntu or Linux Mint:

$ sudo apt-get install joe

To install JOE on Fedora, CentOS or RHEL:

$ sudo yum install joe

To open a text file for editing, run:

$ joe test.log

Loading a large file on JOE is a little bit sluggish, compared to glogg above. It took around 30 seconds to load a 1GB file. Still, that's not too bad, considering that a file is fully editable now. Once a file is loaded, you can start editing a file in terminal mode, which is quite fast.

The memory consumption of JOE is impressive. To load and edit a 1GB text file, it only takes 47MB of physical memory.

If you know any other way to open/edit a large text file on Linux, share your knowledge!

Subscribe to Xmodulo

Do you want to receive Linux FAQs, detailed tutorials and tips published at Xmodulo? Enter your email address below, and we will deliver our Linux posts straight to your email box, for free. Delivery powered by Google Feedburner.

The following two tabs change content below.
Dan Nanni is the founder and also a regular contributor of Xmodulo.com. He is a Linux/FOSS enthusiast who loves to get his hands dirty with his Linux box. He likes to procrastinate when he is supposed to be busy and productive. When he is otherwise free, he likes to watch movies and shop for the coolest gadgets.
Your name can also be listed here. Write for us as a freelancer.

13 thoughts on “How to open a large text file on Linux

    • I tried vim & emacs to edit 1GB file. They easily get choked if you don't have sufficient memory ( > 1GB). Is there any other text editor with big file support (which allows me to edit files bigger than memory)? At least Joe works okay.

      • I get the same behavior, too. What does everyone else consider a big file? We used to have big text files (mostly CDRs) around 3-5GB each. We tried a variety of editors and none of them could open them efficiently. We'd end up just grepping whatever we needed at the time and redirect it to another file.

  1. For editing large files with VIM you can always use the -e option, which starts VIM in Ex Mode (line editor mode). Since in this mode VIM does not render all lines, it uses less memory and is much faster.

  2. ViM does it quite easily if you have free memory. It's my choice as admin (and I do not edit plain text files as user, so it's my choice almost always)

  3. Good info. I have worked with some large password files for labs 2gigs or more. Wish I had known about this earlier. :)

  4. For quick peeks where I'm not sure if what's in the file is actually important, i normally use emacs (though vi variants probably work well here too). If i need to view the whole file, sublime text 2 has treated me well (far better than CLI editors in terms of responsiveness when working with large files for some reason). Then as other people said, the CLI tools work perfectly for all other cases.

  5. In Linux, you open a file with fopen(). You can use open fopen() in Windows too, even if the text books recommend CreateFileEx(). You can read a file with fread() and write to it with fwrite(), until you use fclose(). The C programming language distinguishes between text and binary files for Operating Systems that require it, like Windows, but in Unix there is no difference.

    To edit a file, you would run an editor program.

  6. What's in the text file?

    If it's some sort of line-structured log-like file, I'd probably start with grep, less/more, and similar tools to discern the structure of the information I want to extract, and then probably write a custom filter in some combination of sed, perl, awk, python, and shell to tie them all together. In short, I wouldn't really try to open such a file, especially not one in the gigabyte range.

    You can do the same approach with files that are more structured, perhaps json or ...

Leave a comment

Your email address will not be published. Required fields are marked *