      ANTI-VIRUS SCANNER ANALYSIS BY USING THE "IN THE WILD" TEST SET

                              Marko Helenius

Virus Research Unit, University of Tampere, Department of Computer Science
                 P.O.BOX 607, 33100 TAMPERE, FINLAND
    Tel: +358 31 215 7139, Fax: +358 31 215 070, E-mail: cshema@uta.fi

This paper briefly introduces our methods of testing and the results of a 
test prepared for MikroPC magazine in August 1994. The test was 
performed with DOS-, Windows-, Netware- and memory resident
versions of the scanners by using the "in the wild" test set. I have also 
tried to think what a reader of the results should be aware of.

ACKNOWLEDGEMENTS

Permission is granted to distribute copies of this information, provided 
that the contents of the files and information is not changed in any way 
and the source of the information is clearly mentioned. To republish the 
information permission must be obtained from both Virus Research Unit 
and MikroPC magazine.

INTRODUCTION 

It is too often a myth how an anti-virus tester is working, what viruses 
he/she is using in the tests, what viruses he/she is in possession of and 
what kind of tools he/she is using. This is not however how this should 
be. I believe it should be known to the public, how anti-virus testers are 
working and what viruses they are using in the tests. Also I believe, that 
a tester should admit the lacks of his/her tests to avoid misleading 
information. Also I believe, that co-operation between anti-virus 
researchers is essential to give more resources for fighting against 
viruses. All anti-virus researchers have special knowledge and tools, 
which they are using. Exchanging these would be helpful to all anti-virus 
researchers. Also it would be wise to have advanced virus exchange 
between trustworthy anti-virus researchers. I believe that if all anti-virus 
testers could consider these methods, testing could take a great step 
towards the professional anti-virus testing described by Vesselin 
Bontchev [Bontchev 1994].

To give more exact picture of our work I have briefly presented our 
methods of testing, problems with collecting the "in the wild" test set 
and some things a tester should be aware of. In this paper is presented 
classification of viruses in view of an anti-virus tester, maintenance of 
our virus collection, problems with collecting the "in the wild" test set, 
how we are carrying out the memory resident scanner and boot sector 
virus tests, how we are testing are viruses capable to spread, the results 
of the test performed with the "in the wild" test set and what a reader 
should be aware of when reading the results.

CLASSIFICATION OF VIRUSES

Virus is often called as a self-replicating program, which attaches a self-
replicating copy of itself into other programs. For an anti-virus tester it 
should be obvious, that he/she separates non-viruses from the test set or 
if he/she decides to include Trojan Horses or joke programs in the test 
set, he/she should do a separate test for these and clearly mentions the 
objects used in each part of testing. But what about droppers e.g. 
programs, which are releasing viruses ? Droppers should be excluded 
from the test set although one could say, that these are viruses, because 
these are capable to spread and attach a self-replicating copy of 
themselves into other programs. Droppers should however be 
distinguished from viruses, because searching these by a scanner is 
different from searching actual viruses launched by the droppers. What 
about first generation viruses e.g. original sample files created by virus 
authors ? If the contents of the virus changes so that the later generation 
replicates do not match the first generation virus, the first generation 
virus should be treated as a kind of dropper. Otherwise tests may have 
false results, because some scanners may detect the first generation 
viruses, but undetect the later generation replicates. To avoid this kind 
of problems, it is wise to avoid using first generation viruses in the tests.

MAINTENANCE OF A VIRUS COLLECTION AND A TEST TREE

In the Virus Research Unit viruses are preserved in two large directory 
trees corresponding with each other. Samples of each virus is stored in 
the leaf directories of the trees. The other tree includes original sample 
files, even first generation viruses may be included, and the other files, 
which are infected with the original viruses. This latter directory tree, 
called test tree, is used for testing anti-virus products. This division is 
done to avoid first generation viruses to be included in the test sets and 
on the other hand to restore the original sample files.

Whenever we receive a new set of viruses we do the following things. 
First of all we will separate Trojan Horses, joke programs, droppers and 
other non-viruses from the new set of viruses. Only obvious cases are 
separated at this stage. After this duplicates and already existing viruses 
are separated by a specific tool called VIRSAMPL. This specific tool 
checks weather there are among new set of viruses already existing 
viruses. Mostly there are and new samples are moved into the same 
directory as existing viruses and duplicates are deleted. Next thing to do 
is to check weather the rest files are capable to spread. This spread 
testing is done automatically by a specific boot disk called "Spread test" 
(see section below). Original samples of viruses capable to spread are 
moved into the collection and infected files are restored into the test 
tree. New viruses are moved into new subdirectories by a specific tool 
called NEWBAT. NEWBAT creates a batch file, which automatically 
does the directory creation and copying process. Rules for this copying 
process must be defined in a separate file.

COLLECTING THE "IN THE WILD" TEST SET

For a virus to be included "in the wild" test set, it must have been found 
in the "field" at least once. This is not however as obvious as it sounds. 
How do we know, that a virus has been found in the field at least once ? 
Someone must have reported to some anti-virus researcher, that the 
virus has been found in the field. There is still one problem. How do we 
know that someone has reported the virus to some anti-virus researcher 
? One solution is to use the Joe Well's list [Wells], which includes 
viruses, which have been reported as found in the field according to 
main anti-virus researchers. It does not however contain all the viruses 
found in the field, because all cases are not reported to Joe Wells. For 
example we have in Finland many viruses found in the field, which have 
been reported to anti-virus researchers and/or to Central Criminal Police, 
but are not in the Joe Well's list. I have also reports from other anti-virus 
researchers of viruses found from the field and which are not in the Joe 
Well's list. However those viruses mentioned in the Joe Well's list should 
at least be included in the test set. My solution was to include besides 
those viruses mentioned in the Joe Well's list viruses found in Finland 
and ask for comments from anti-virus researchers on viruses found in the 
field. (Thank you for those of you, who co-operated.)

The problems were not behind after this solution, they were just about to 
begin. Non of the "in the wild" listings had exact information, which 
variants of viruses were found in the field. Sometimes the exact variant 
could be identified directly, but in most cases further examination was 
needed. Luckily most of the viruses found in the field in Finland were 
available to Finnish anti-virus researchers and I could be certain of the 
correct variant. After the basic work was done I had to compare several 
sources of information between each other to determine, which variant 
of the virus was "in the wild". I compared between each other listings of 
our virus collection, listings of CARO's naming standards, information of 
viruses provided with anti-virus products, listings of viruses I had 
already determined as "in the wild" and several listings of viruses "in the 
wild" (Joe Well's list, listings from Virus Bulletin and listings from other 
Anti-Virus researchers). In most cases this comparing process was 
producing results and I could almost certainly identify the correct 
variant, but this was a result of lot of work and time efforts used. Still I 
cannot be absolutely certain, that all variants were chosen correctly.

BOOT SECTOR VIRUS TESTS

It would be too hard to do the boot sector virus testing by changing a 
diskette between each different virus. This is why we are using a 
program, which copies diskette images from files to diskettes 
automatically. To determine, which image was originally from which 
type of diskette we have determined to name files as follows: *.IMG 
means 5.25" DD, *.IBD means 5.25" HD, *.IHD means 3.5" HD and 
*.IDD means 3.5" DD. After the image has been copied path and file 
name of the image file is added after test reports and then scanners are 
executed. All of this is done automatically by using a batch file, which 
does the image copying and scanning process for each tested image file 
separately. 

TESTING MEMORY RESIDENT SCANNERS

We have decided to perform the memory resident scanner tests of file 
viruses by copying infected files while a tested scanner is memory 
resident and set to check copied files.  First of all there must be a 
program or a batch file, which copies all sample files included in the test 
set into another location. Commands like "XCOPY C:\TESTSET 
D:\NOTFOUND /S" are not enough, because most scanners denies the 
access of the command and copying stops after first infected file has 
been found. It is not however difficult to create such batch file. 
Command "DIR /S /B>TEST.BAT" or commands "ATTRIB -A /S" and 
"ATTRIB /S>TEST.BAT" are enough to create the listing of files into 
the batch file. Now each line of the batch file must be edited to 
correspond form "XCOPY [FILE NAME] [TARGET FILE NAME] 
<D.DAT" and the file "D.DAT" must include letter "F". (If the target file 
name includes a file name, otherwise "D".) Now the created batch file 
copies each file separately and the copying does not stop when a virus is 
found. We have also a more specific tool, called "YCOPY", which 
creates the batch file automatically. 

Boot sector virus tests of memory resident scanners are carried out by 
attaching infected disks with "DIR"-command. The image copying 
process is equivalent with the process described in the previous section

Now there is still one problem especially, if there is a large number of 
sample files or disks to check. Most  memory resident scanners causes a 
prompt to appear whenever they have found a virus. A user of the 
product must press some key or key combination to continue. To 
automate this we have a memory resident monitoring program, which 
follows, what happens on the screen. Whenever the monitoring program 
notices, that the memory resident scanner prompts to press a key, the 
monitoring program "pushes" the key or key-combination automatically. 

Now with these tools the testing is possible even with large amount of 
viruses. And how do we see, which viruses were found and which were 
not ? This is quite easy, if the original path and file name is restored 
while copying. This can be directly seen from the target directory tree, 
which shows viruses gone through e.g. viruses which were not found.

TESTING ARE VIRUSES CAPABLE TO SPREAD

To automate testing is a virus capable to spread we have so far 
developed the following concept. First a computer is booted from a 
floppy disk (called "Spread test"), which is write protected and has 
AUTOEXEC.BAT edited for this specific use. The computer loges into 
the Virus Research Unit's local network and gets first sample file from a 
specific directory, loges out, generates target files, calculates checksums, 
executes the sample file, executes the generated target files and does this 
three times by changing date and time between each run. After this 
computer is automatically cold booted from the same floppy disk. Now 
it is tested, weather checksums have been changed and if there are 
changed checksums, files are moved into the network into another 
location in path corresponding the original file name and the computer is 
automatically booted again. This procedure continues until there are no 
more files to check. The drawbacks of this concept are so far, that the 
system cannot so far automatically recover from "hanging" or damage 
the sample files may cause. We are however developing more advanced 
concept, which could automatically recover from hanging and damage 
and so we could completely automate the testing process. All theory and 
hardware for this concept is ready or available, but the software for this 
is still under development.

POLYMORPHIC VIRUS GENERATION

To automate polymorphic virus generation, we have the following 
concept. First a batch file is executed from a write protected floppy disk. 
The batch file calls a program, which generates target files of random 
sizes (number of files generated and size range can be given as a 
parameter for the program). After this checksums of generated files are 
calculated, sample file is executed and target files are executed. After 
this computer is cold booted, checksums are verified and changed files 
are moved into different location.

REPORT FILE CREATION

I believe, that it should be known to the public, what was actually tested 
and how did we came into the results in a particular test. So this is why 
we have always made a listing, which shows which sample files were 
found by which product. To do this we are first using awk-scripts to 
organise the report files into analogous format. After this we use a 
specific tool called "VIRLIST", which unites the report files.

RESULTS OF THE TEST

The test set included 51 boot sector viruses and 182 file viruses found in 
the field according to anti-virus researchers. Table 1 introduces the main 
results of the different sections of testing. Mark "I" means that the 
testing procedure included both boot sector and file viruses and mark 
"II" means that the testing procedure included only file viruses. "Scan" 
means normal scanning process and "Heuristics" heuristic scanning 
process. Results of heuristics includes viruses detected by heuristic 
scanning besides viruses detected by normal scanning. "TSR" means a 
test of memory resident scanners, "Windows" means a test of Windows 
versions and "NLM" means a test of Netware versions. In most cases 
there were several samples of the same virus. If a product could detect 
only part of the samples, I decided to count averages of these samples 
although one can say, that a partly detected virus may cause trouble for 
a user, because the undetected samples may cause reinfection of the 
virus. I have not performed any further descriptions of these results, 
because the results implement only a snapshot and an overall impression 
of the products' performance.

Product                  Scan I Heuristics I TSR I   Scan II Windows II NLM II
Thunderbyte 6.20         86,04% 12,54%       77,88%  -----   -----      -----
Anti-Virus Toolkit 6.65  97,42% -----        90,56%  96,70%  96,70%     96,70%
F-PROT Professional 2.13 97,00% 0,40%        82,20%  96,15%  86,81%     77,17%
Sophos Sweep 2.12        95,69% -----        -----   95,03%  95,03%     95,03%
IBM Anti-Virus 1.06      94,32% -----        No test 92,73%  92,73%     91,63%
McAfee Scan 2.1.0        90,57% -----        83,70%  89,02%  89,02%     88,73%
Norton Anti-Virus 3.0    75,76% -----        75,76%  81,06%  81,06%     No test
Central Point AV 2.0     74,33% -----        46,68%  -----   No test    No test
Microsoft Anti-Virus     52,97% -----        46,39%  -----   No test    -----

TABLE 1: DETECTION PERCENTAGES

Table 2 introduces polymorphic virus detection by different products.

Virus name     Toolkit F-PROT Sweep  TBAV  Scan  IBM   NAV   CPAV  MSAV  Total
TPE.Bosnia     1001    214    1001   1001  881   999   232   0     0     1001
TPE.Girafe     1525    1425   1525   1505  661   1514  1500  0     0     1525
MtE.Pogue      1108    1108   1108   1108  1108  1108  1108  1000  1108  1108
Phantom1       1181    0      1176   1183  0     0     1     0     0     1183
Smeg.Queeg     1105    1109   675    1074  0     1110  0     0     0     1110
Smeg.Pathogen  1301    1304   1304   1267  1233  1304  0     0     0     1304
MtE.Coffeeshop 1032    1031   1032   1032  1014  1032  1014  946   887   1032
Total          8253    6191   7821   8170  4897  7067  3855  1946  1995  8263

TABLE 2: DETECTION OF POLYMORPHIC VIRUSES

The names of products are shortened as follows in the table 2:
Toolkit = Dr. Solomon's Anti-Virus Toolkit 6.65
F-PROT = F-PROT Professional 2.13
TBAV = Thunderbyte Anti-Virus 6.20
Sweep = Sophos Sweep 2.12
Scan = McAfee Scan 2.1.0
IBM = IBM Anti-Virus 1.06
NAV = Norton Anti-Virus 3.0
CPAV = Central Point Anti-Virus 2.0
MSAV = Microsoft Anti-Virus (MS-DOS 6.2)

DISCUSSION AND CONCLUSIONS

A reader of these results should be aware of the following things, which 
I believe should be noted while examining the results. First of all it 
should be noted, that the performance of checksum calculation programs 
was not tested although there is a checksum calculation program 
included with all tested products. Also products' disinfection capabilities 
were not examined. In addition it should be noted, that the tests were 
not carried out while viruses were memory resident although this is often 
the case, when a computer is infected with a virus. Also it should be 
noted, that all viruses "in the wild" were not included in the test set, 
because only viruses we know as being "in the wild" were included. In 
addition we might have done a mistake while checking correct variants 
of  the viruses. Also it is always possible, that someone has given us 
misleading information although we have tried to do our best to verify 
the information given us whenever possible. Also it should be noted, that 
we did not try to measure how common each virus is and so the 
percentages do not directly measure the actual risk level of infection. 
Instead the percentages just show, how many percent of the viruses used 
in this test the products could detect. We have done our best to exclude 
non-viruses, droppers and first generation viruses from the test set. 
However we might have done mistakes because of tight schedule and so 
it is possible to have non-viruses included although I believe that there 
are only few such mistakes. Also it should be noted that we did not try 
to check weather a product can reliably detect a virus e.g. we did not 
count cases, where a product did not detect all replicates of a same 
virus. Because of the drawbacks mentioned above it should be noted, 
that these results give only an overall impression of the tested products' 
performance. However it should be noted that often also other tests 
concerning anti-virus products have most of the above drawbacks. 

Regardless of the drawbacks I believe, that there are some advantages in 
this test. We have achieved to include most memory resident scanners, 
Windows versions and Netware versions in the tests. Another advantage 
is, that because of lot of work and time efforts used, I believe that most 
common viruses were included in the test set. In addition this test has 
advanced cross-references, which can be directly read into spreadsheet 
programs. The cross-references clearly show, which viruses were 
detected by which product and by which name. 

REFERENCES

[Bontchev]   Vesselin Bontchev, "An Analysis of Anti-virus Scanners", 19.7.1994       Available electronically via anonymous ftp as
             ftp.informatik.uni-hamburg.de:
             /pub/virus/texts/tests/vtc/tests-01.zip

[Wells]      Joe Wells, "PC Viruses in the Wild"
             Available electronically via anonymous ftp as
             ftp.informatik.uni-hamburg.de:
             /pub/virus/texts/viruses/wild9407.zip

