=======================================================
COMPRESSING YOUR EXECUTABLES --- SOMETHING FOR NOTHING?
=======================================================
by Dickford Cohn
Ah yes, the current "rage" in PC-land: EXECUTABLE FILE COMPRESSION! Does it work? Yes! Very well! Is it "something for nothing?" Well, you be the judge — in my experience, it is VERY useful. This article is intended to "corral", in one place, most of the "bits and pieces" of what is known about these programs and to offer performance comparisons and operating tips gleaned from those who have used them. This article centers upon the THREE most common methods available to us — those that I've used and tested.
–WHAT IS EXECUTABLE FILE COMPRESSION? It is simply a method by which an EXECUTABLE file (.COM or .EXE) is made small- er on disk (outwardly similar to PKZIP). If you have a lot of executables, you can save substantially on disk space. When a file is "compressed" a small addi- tion is made to the code that UNCOMPRESSES the executable to "full size" when it is loaded into RAM. It's that simple (yeah, the programming is a LOT more involved than that!)
–IS EXECUTABLE FILE COMPRESSION SOMETHING I CAN USE? It depends on how you use your system and what you have on it. For example, if your PC emulates your office PC in that you have 50-60 executables and you pri- marily have spreadsheet/word processing programs on your hard disk, an execut- able file compressor would probably not be worth the expense of a download. On the other hand, if your system has LOTS of executables, executable file com- pression is quite effective in re-capturing valuable disk space. In my own case I have almost 950 executables that, at normal size, take up almost 25Meg of disk space. I have compressed over 500 of these executables, re-capturing almost 9Meg of disk space — representing about 15% of my TOTAL OCCUPIED DISK SPACE! TWO IMPORTANT NOTES: (1) Executable file compression is NOT a replacement for other disk management techniques (i.e. good housekeeping): It IS an effective tool to add to your arsenal. (2) Executable file compression is NOT A GOOD IDEA in the office setting — unless YOU are SOLELY responsible for your PC: In the hands of the uninitiated, these programs can be dangerous.
–IS EXECUTABLE FILE COMPRESSION DANGEROUS? ANY program that can OVERWRITE an existing file is dangerous! You MUST have a BASIC UNDERSTANDING of what you are doing and employ the standard safeguards that ANYONE would employ (i.e. backups) when dealing with programs that have this capability. With that in mind, it must be said that ALL the programs des- cribed here in detail are thoughtfully designed and have certain safeguards built in. However, there is NO EXCUSE for NOT READING THE DOCUMENTATION — it is BRIEF and CONCISE.
=====================================
ADVANTAGES OF COMPRESSING EXECUTABLES
=====================================
There are TWO distinct advantages to compressing executable files:
1. You can achieve a substantial savings in disk space by using these util- ities: Typically, you can re-capture 35-40% of the disk space now occupied by your executables. You have to determine if the OVERALL space gained is worth the effort.
2. A less-heralded but equally significant benefit of executable compression
is that compression (a form of encryption) offers some protection against
viruses that overwrite program code. Since the "file header" is relocated,
the virus can't find it (to attach itself); hence the virus cannot over-
write code. However, the executable is still "externally infected" --- it
CAN load the little critter into RAM, ready to "pounce" on some uncompress-
ed executable. At least, your executable is not TRASHED.
======================
GENERAL TIPS FOR USAGE
======================
1. DO NOT COMPRESS DOS's COMMAND.COM, or ANY CRITICAL DOS FILES --- if you don't know which DOS files are CRITICAL, don't compress ANY OF THEM. (Watch for my upcoming article, HARD DISK MANAGEMENT TIPS, which will include a section on DOS files you can "dump" or replace with "modern" equivalents: Saves about 300k on your disk!)
2. If you choose to compress executables at all, pick ONE method and stick with THAT ONE. The confusion caused by "mixing 'em up" is not worth it.
3. When compressing executables for the first time, or experimenting, create a separate directory and COPY your "target" executable to it (this precludes damage caused by "typos" at the command line and such --- if you screw up, all you have to do is RE-COPY the file and try again.) When you are satisfied with the result, RENAME the ORIGINAL executable, and copy the COMPRESSED executable to its original directory. Now, TEST the "new" executable --- call up your program and see if ALL of its functions work. I mention this because SOME programs will compress just fine (i.e. with NO WARNINGS from the compression program) and yet they MAY NOT work properly, if AT ALL.
4. Some programs have a companion "configuration" program that WILL NOT WORK on a compressed executable. Just UNCOMPRESS the executable, re-configure (via the config program) what you want and then RE-COMPRESS the executable.
=================
WHAT'S OUT THERE?
=================
At this writing, I am aware of EIGHT executable file compression programs:
- AXE …A program from Systems Enhancement Associates. This is
a commercial software package ($65) that, outwardly at
least, offers no significant advantages over other methods
(see COMPARISON) and is NOT reviewed here.
- EXEPACK …A MicroSoft product, NOT generally available but packaged
with certain software (MASM 5.1). Although, apparently the
FIRST such program of this type (1986 copyright), it is NOT
in the same league as those reviewed here, although you are
LIKELY to encounter this method with some executables.
- DIET …A program by the author of LEXEM (functions like PKZIP and
apparently in wide usage in Japan). DIET is reviewed here.
- LEXEM …Not currently distributed in the U.S. (Likewise, you are
NOT likely to encounter files compressed with this method)
- LZEXE …By Fabrice Bellard; the first generally available program of
this type — reviewed here.
- PKLITE …Shareware from PKWARE ($46) — reviewed here.
- SHRINK …An experimental method, not in the same class as those re-
viewed but interesting to those who may want to delve into
the "mysteries" of compression.
- TINYPROG …Shareware from Tranzoa Associates. Only slightly more eff-
ective than EXEPACK: No UNCOMPRESS available. Not reviewed.
====================
PROGRAM DESCRIPTIONS
====================
+————+ The "newest" of the three tested, by Teddy Matsumoto, the author | DIET10.EXE | of LEXEM an "archive" program used in Japan. This program app- +————+ ears well written -BUT- by the authors own subtle admonition, it
is experimental. It SEEMED to work slightly better than the
other two -BUT- and this is a big BUT: IT DOES NOT CHECK FOR
EXECUTABLES THAT USE OVERLAYS --- IT JUST COMPRESSES THEM!
YOU WILL NOT BE ABLE TO RESTORE THE EXECUTABLE WITH THE '-R'
SWITCH, EITHER --- THE FILE IS DESTROYED! (That's WHY you
maintain back-ups). The program DOES ALLOW you to make back-ups
however, so you should be safe. When I attempted to compress
a 1 Meg+ executable (that I knew used overlays), the error that
came up was "insufficient memory" (more about this later).
DIET also compresses using wildcards and WILL COMPRESS virtually
any file EXCEPT one that is already compressed by another
program.
PRICE: The current version on EXEC-PC (DIET10.ZIP) is **FREE**
+————+ Developed by Fabrice Bellard (current version is 0.91, backward | LZEXE.EXE | compatible with version 0.90). This program was the FIRST +————+ generally available executable file compressor — and is the
most widely used (see SUPPORT PROGRAMS). It also is programmed
with some forethought: For example, it can recognize the
MicroSoft EXEPACK compression method in executables; it can
UN-compress EXEPACK'd files and RE-compress them with it's own
method (the savings is significant). It's error prompts work
as advertised -BUT- they're too polite! For example, if you
decide to go ahead and compress an executable that uses over-
lays, you'll NEVER restore it with the un-compressor UNLZEXE.EXE
However, since LZEXE DOES NOT OVERWRITE your ORIGINAL FILE, you
are safe! LZEXE copies your original file to an ".OLD" exten-
sion: You simple erase the "bad .EXE" and RENAME the ".OLD"
file to an ".EXE" extension. The original program's prompts are
in French (there is a program in English --- ENGLZEXE.ZIP): I
would recommend LZESHELL to sensibly use LZEXE. LZEXE also com-
presses .COM files, but it first converts them to .EXE files and
THEN compresses them. This measurably slows down the process
AND you CANNOT restore "converted" .EXE to .COM files (the .OLD
file is STILL created, though). This program comes with LZEXE
(the compressor), UPACKEXE.EXE (unpacks EXEPACK'd files) and
COMTOEXE.EXE (converts .COM files to .EXE). You MUST also have
UNLZEXE.EXE, by Kou Kurizono, to UN-compress LZEXE'd files!
PRICE: The current version on EXEC-PC (LZEXE91.ZIP) original French
version -OR- (ENGLZEXE.ZIP) English version, are **FREE**
+————+ Developed by PKWARE (the purveyors of PKZIP, et al) this program | PKLITE.EXE | is a textbook example of "completely developed" software. +————+ Unlike the other two examples, there are no "cliffs" to fall
over; no unforgiving "surprises". Performance was slightly less
than the other two overall --- the difference is nominal. The
elements I liked best are: This is a COMPREHENSIVE package, fea-
turing a logical selection of command line options; it IS
INTUITIVE; -AND- unlike the other two, it could COMPLETELY RE-
CONSTRUCT an executable with overlays that it had COMPRESSED!
This program compresses (and UN-compresses) .COM and .EXE only.
PKLITE WILL NOT COMPRESS Windows Executables (A SAFETY FACTOR).
PRICE: The current version on EXEC-PC (PKLTE103.EXE) costs **$46**
===================
SUPPORTING PROGRAMS
===================
The following is a PARTIAL listing of programs that provide support for the compression programs above (files on EXEC-PC):
ù LZESHL21.ZIP - Excellent English Language shell for LZEXE 0.91 ---
enables and TRANSLATES ALL functions of LZEXE from
within. By Pete Petrakis **FREE**
ù DRX100.ZIP - User-configurable shell, provides directory/subdirectory
display of COMPRESSED/UNCOMPRESSED executables (AXE,DIET
PKLITE, EXEPACK, LZEXE and TINYPROG). Allows you to sel-
ect COMPRESS/UNCOMPRESS method: Offers "point-n-shoot"
file selection. By Raymond T. Kaya **FREE**
ù CHK4.ZIP - Re-directable compressed file reporter, with many report
options. Now "sees" ALL compressed executables on the
entire drive, by subdirectory. By John Land **FREE**
ù UNLZEXE5.ZIP - REQUIRED FOR LZEXE. UN-compresses both versions of LZEXE
(Vers .090/.091). By "Kou" Kurizono **FREE**
Note that there are OTHER programs available, but these are the ones I've
used, so I can attest to their reliability: They're all EXCELLENT!
=======================
PERFORMANCE COMPARISONS
=======================
The test data shown in the table(s) below are relative, and by no means, exhaustive: Timing was performed with a Turbo C++ shell that converts standard "timer ticks" to fractions of a second. I must stress that since you're likely to use these programs only once on a given executable, execution TIME is NOT A SIGNIFICANT FACTOR. Programs were also tested to see if the various safeguards outlined in the DOCS actually worked: I'm happy to report that they performed flawlessly in ALL cases.
File/Original Size
----------------------- ....Compressed with.... ...UNcompressed with..
QWIK.EXE/8873 Bytes DIET LZEXE PKLITE DIET UNLZEXE PKLITE
-----------------------
COMPRESSED SIZE 4937 5070 5164 -- -- --
TIME (seconds) 1.8 1.3 1.9 1.9 2.8 1.9
SIZE REDUCTION: 44% 43% 42%
============================================================================
----------------------- ....Compressed with.... ...UNcompressed with..
FW.EXE/26510 Bytes DIET LZEXE PKLITE DIET UNLZEXE PKLITE
-----------------------
COMPRESSED SIZE 20236 21131 20970 -- -- --
TIME (seconds) 1.9 2.6 2.8 2.2 3.8 2.8
SIZE REDUCTION: 24% 20% 21% ============================================================================
————————- ….Compressed with…. …UNcompressed with..
CSHOW1.EXE/89904 Bytes DIET LZEXE PKLITE DIET UNLZEXE PKLITE
————————-
COMPRESSED SIZE 53584 52211 55572 -- -- --
TIME (seconds) 3.0 8.6 9.3 3.1 5.3 4.5
SIZE REDUCTION: 40% 42% 38%
PERFORMANCE COMPARISONS (Continued)
————————- ….Compressed with…. …UNcompressed with..
TURBO.EXE/156321 Bytes DIET LZEXE PKLITE DIET UNLZEXE PKLITE
————————-
COMPRESSED SIZE 107350 110398 107765 -- -- --
TIME (seconds) 4.5 16.8 22.9 4.1 6.1 6.2
SIZE REDUCTION: 31% 29% 31%
SUMMARY:
Average Time:(Secs/Kbyte) .0398 .1041 .1310
Average Size Reduction: 34% 33% 32%
==============================
As you can see by this modest "test", ALL of these units performed about the same --- differences are negligible. Overall, DIET was nominally "quicker" AND compressed more "tightly" (for you performance enthusiasts). It WAS INTERESTING to NOTE that the effectiveness of the compression is apparently related to the development method (programming language) of the executable. Note, for example, the DIFFERENCE in the SIZE REDUCTION between QWIK.EXE and FW.EXE. I KNOW that FW.EXE is programmed entirely in ASSEMBLER, while QWIK.EXE is programmed in Turbo C++: I DON'T KNOW enough about the technical aspects of compression to comment intelligently on why this happens. One other item was noted: Compressed executables do SEEM to take a slight bit longer to do their thing --- anywhere from a fraction of a second to three seconds: This on a 16Mhz PS/2 --- the differential is GREATER on slower machines: (When tested on an IBM "Turbo" XT (8Mhz) --- execution time is proportionately slower). During the tests, PKLITE was used with the "create backup file" switch ON (strangely, it seemed to work faster that way). DIET was used BOTH ways (create backup/just overwrite) with no apparent difference in time.
======================
COMPARISON OF FEATURES
======================
DIET LZEXE PKLITE
------ ------ ------
û-Tests for/Warns of Executables
that use OVERLAYS NO(1) Yes(1) Yes(1)
û-Indicates File Already Compressed NO(2) Yes(2) Yes(2) û-Identifies Compression Method NO(2) Yes(2) Yes(2) û-Safeguards Against Compressing
WINDOWS Executables NO NO Yes
û-Restores Compressed Executables
that use OVERLAYS NO NO Yes
û-Automatic Back-Up of Original NO Yes NO û-Back-Up Option (Switch) Yes NO Yes û-Compresses Files OTHER than
Executables Yes(3) NO NO
û-Compresses .COM and .EXE Yes Yes(4) Yes û-Compresses ANY SIZE File NO(5) Yes Yes û-UN-compresses Files it
Compresses Yes NO(6) Yes
û-Support Limited Limited FULL û-Documentation Good(7) Good(7) Excellent
NOTES………………………………………………………………. (1) Both PKLITE and LZEXE warn of files using overlays: DIET just OVERWRITES
the TARGET file --- you MUST SPECIFY the "create backup" ('-O') option
on the command line. This is a potentially fatal flaw with DIET 1.0!
NOTES (Continued)……………………………………………………. (1a)K. Okubo (on CIS) insists that DIET DOES NOT COMPRESS executables that use
overlays --- that has NOT been my experience.
(2) DIET and PKLITE simply state "file cannot be compressed" — LZEXE can
detect files compressed with EXEPACK, un-compress them and re-compress them with its own method: A valuable option with some commercial software. PKLITE now has a little utility that WILL identify its own compression method --- NONE of these utilities can identify compression methods OTHER than their own. (The shell DIRX takes up the slack here).
(3) DIET can compress ANY file, executable or otherwise. HOWEVER, there is no
visual indicator (e.g. a .DIE extension) to tell you that they ARE com- pressed (they are NOT usable in their compressed state --- they must be "manually" uncompressed, first). I'm hard pressed to come up with a use for this option: I suppose you COULD compress entire subdirectories that were seldom used, but this could get to be tricky; particularly if you moved the files to another directory or to diskette. PKZIP is much more useful in this regard than DIET -and- more intuitive.
(4) LZEXE first must convert .COM files to .EXE files, slowing down the process
of compression (I would "guesstimate" that it would be roughly double the time shown in the "time trials" but remember that .COM files are limited to 64k in size, so again, time is NOT significant). Also, you must remember that you cannot convert COMTOEXE'd .EXE files back to .COM!
(5) DIET apparently differs from the other two in that it seems to bring the
ENTIRE target file into RAM --- if the file size exceeds available memory, DIET produces an "insufficient memory" message and does not execute. Note that PKWARE's file WHATSNEW.103 offers another potential explanation.
(6) In order to UN-compress LZEXE'd files, you MUST use UNLZEXE.EXE (see
SUPPORTING PROGRAMS), which is NOT bundled with LZEXE.EXE
(7) Documentation runs the gamut: The DOCS for DIET are GOOD but I don't
feel they adequately warn of the pitfall of compressing files with overlays: In fact, the DOCS are ebullient and enthusiastic (I would be, too, if I had written the program). LZEXE's original DOCS are in French, so they are of limited value, at best. There IS an ENGLISH version on EXEC-PC that you may want to download. PKLITE's documentation is the usual THOROUGH job, filled not only with specifics about PKLITE, but also file compression in general and even compares PKLITE to LZEXE. It is understandably biased: More significantly, it is VERY informative.
SUPPORT - Support for DIET consists of a CompuServe ID (in the documentation)
to which you can address correspondence (E-Mail). To the best of my
knowledge, you can't contact the author of LZEXE at all. PKWARE
provides FULL SUPPORT for all its REGISTERED software and operates
its own BBS. There are a considerable number of FREE/SHAREWARE
packages that offer additional support for ALL these programs.
========================
A WORD ABOUT OVERLAYS...
========================
Well, you've seen the mention of "...executables that use overlays" throughout this text --- THAT IS INTENTIONAL: Quite a bit of today's commercial software have executable files which use overlays. OK...what are OVERLAYS? Overlays are SECTIONS of a program's function code, which are "read" into the executable "shell" as needed by the program. Confused? Let's use an analogy: Think of your executable as the HANDLE of a SOCKET WRENCH set and OVERLAYS as being the actual SOCKETS: You simply "switch" the sockets (over- lays) in and out as needed to perform the various functions you need. While all of the functions you might want to perform are at your fingertips, you can only perform ONE AT A TIME (i.e. you can use only ONE socket at a time). This is done primarily so that you have enough room in RAM to create/edit a file -AND- load your executable. Additionally, some programs are so large that not ALL of the program could be loaded into the RAM normally available on your system (e.g. Certain spreadsheet programs use overlays extensively --- actually the spreadsheet you're working on IS AN OVERLAY ITSELF.) When a file
A WORD ABOUT OVERLAYS (Continued)
compressor comes along, its "algorithms" determine that the file space in the "shell" is mostly EMPTY space and proceed to re-write the file header to the "correct" file size: A FATAL nicety! Now, there is NO ROOM LEFT for the "overlay" --- i.e. the spindle for the socket is GONE! (Yeah, I know this isn't a PERFECT analogy, but you get the drift.) FORTUNATELY, with two of the three programs mentioned (LZEXE and PKLITE) you are adequately WARNED that the executable uses overlays BEFORE you can proceed.
=============================
WHAT ELSE TO WATCH OUT FOR...
=============================
Some executables (sometimes called "runtime modules") have the ability to actually change their size while in memory --- they write additional code to THEMSELVES in order to accomplish certain tasks or to configure themselves according to options supplied by you. NONE OF THE ABOVE-MENTIONED PROGRAMS warns you of such executables (I have no idea how they could be programmed to do this --- that is WHY you EXPERIMENT in a TEST directory): The file will simply be "hashed"! This does not occur in all cases, but it DOES occur with TWO specific files I'll bet many of you have: (1) BRUN41.EXE by MicroSoft, bundled with all versions of CompuServe's CSHOW, and (2) GRASPRT.EXE, a "GRASP" (.GL) file "animator" from Paul Mace. There are OTHERS, such as MicroSoft's CV.EXE (CodeView --- part of MASM 5.1): WINDOWS "executables" should NEVER be compressed --- this includes Versions 3.0, WIN286 and WIN386.
Also, you've noticed mention of EXEPACK. This is a program copyrighted by
MicroSoft in 1985-86 (probably the FIRST executable file compressor). This
program is used on certain executables in some commercial software and the
EXEPACK.EXE program itself is bundled with some software (MASM 5.1). The
purpose of EXEPACK is not clear: The explanation on Page 322 of "MicroSoft
CodeView and Utilities Guide" is curt and misleading: EXEPACK will NOT
compress executables using overlays. Anyway, this is NOT a free-standing
program and is NOT nearly as effective as those mentioned here. Only
LZEXE can identify files compressed with EXEPACK and, at your option, re-
compress them. DIET and PKLITE do not compress EXEPACK'd executables.
However, you CAN use UPACKEXE.EXE (w/LZEXE) to un-compress them first, and
then re-compress them with DIET/PKLITE --- with apologies to M. Bellard.
==================
HOW TO USE THEM...
==================
Obviously, you MUST have the desired utility in your ROOT directory or on your PATH. The following command line syntax is SUGGESTED if you're new to this or experimenting:
- —————
TO COMPRESS…
- —————
DIET - C:\>diet -o[output filename] [input filename]
(compresses a COPY of INPUT filename to OUTPUT filename ---
original is not altered.) [NO space between "-o" and filename]
NOTE that position of INPUT/OUTPUT filename is REVERSED
LZEXE - (using LZESHELL interface)
C:\>lzeshell [filename]
(backup copy of original executable is AUTOMATIC, w/.OLD exten-
sion. Not necessary to specify ext if .EXE/MUST specify if .COM)
PKLITE - C:\>pklite -b -n [filename]
(create BACKUP file w/.BAK extension [-b] and NEVER compress an
executable that uses overlays [-n])
- —————–
TO UNCOMPRESS…
- —————–
DIET - C:\>diet -r [filename]
(Restore file compressed with DIET --- DIET's DOCS call this
command "Retrieve")
LZEXE - C:\>unlzexe [filename]
(Uncompress file compressed with LZEXE - using UNLZEXE and
create BACKUP of compressed file w/.OLZ extension)
PKLITE - C:\>pklite -x [filename]
(eXtract file compressed with PKLITE)
NOTES: 1. These commands are recommended for use in experimenting with these
programs --- you are by no means limited to them. There are others
included in each program, so read the documentation for proper use.
2. Remember to DELETE the BACKUP files when you're satisfied with the
results --- the idea, after all is to MAKE ROOM not USE MORE OF IT.
3. Note that PKLITE creates BACKUP files with the .BAK extension ---
you should keep track of these files because it is the SAME ext-
ension DOS (and some utilities) use --- it could get confusing.
========================
OK...WHICH ONE DO I USE?
========================
Choosing the RIGHT compression method is a matter of personal preference: They are ALL GOOD! PKLITE seems to offer the most due to a variety of factors: Excellent, intuitive and SAFE design; the promise of ongoing development; total support (for registered users); variety of features, etc. PKWARE is the outfit that has made "ZIP" a household word: They take their products seriously and they DO KNOW something about "compression". This DOES NOT mean to imply that either Mr. Bellard or Mr. Matsumoto takes his product any LESS SERIOUSLY — the quality of BOTH products proves otherwise. Why pay 46 bucks for a product when you can get it for FREE? Well, SUPPORT is a two-way street: Whether we want to believe it or not, SOMEBODY paid for the development of all this great software. Mr. Bellard and Mr. Matsumoto BOTH paid with their time and talent: Considerable, in BOTH cases. PKWARE also PAID with their time and talent, with the slight prospect of recouping these costs through license fees. Nobody works for NOTHING! I think PKWARE also lent some credence to this heretofore obscure "niche" in PC utilities with their entry into the "fray" — something to consider, eh?
==========
CONCLUSION
==========
To answer our original question: Yes, we get something for remarkably little effort. I was immediately struck by the fact that ALL of the programs are well-crafted, possessing common sense and foresight: They can be very useful tools for the average PC user. The companion utilities described in SUPPORTING PROGRAMS also deserve quite a hand for their quality and development skill. They make life a lot easier for those of us prone to using executable compres- sion. Very thoughtfully programmed stuff that, I'm sure, took considerable effort, time and skill. The mere fact that ALL of this excellent programming is available to us, belies the oft-noted selfishness claimed to be "today's credo". IT AIN'T SO! These folks deserve our respect and thanks!
In conclusion, it could be stated that "compressing" your executables is an idea whose time has come. You will definitely save disk space and might even defer the three to five hundred dollar outlay for ANOTHER disk drive. It's worth considering when you have available the utilities discussed here.