=========================
File 5PROTOCO.TXT
AV Product Test Protocol:
=========================
Formatted with non-proportional font (Courier)
Remark: no changes since last test (2001-10).



This document specifies the test procedures applied to test the 
precision of detection as well as the reliability of detection of 
PC-based boot, file and macro viruses. Moreover, test procedures 
for determining detection of packed viral objects and non-viral
malware are also described. Where relevant, details concerning 
differences against previous VTC tests (esp.2000-04/08) are given.


1) Hardware and System Software used:
-------------------------------------
Test "2001-10" installation differs from last test (2001-04) essentially 
in updated testbeds (which were frozen on April 30, 2001), and that it
tests macro and script viruses/malware only. 

Again, the detection of viral code in packed (file and macro) objects
was tested for the set of In-the-Wild viruses, including 6 popular
packers (ZIP, LHA, ARJ, RAR, WinRAR, CAB). Moreover, a set of non-malicious 
objects was used to determine the ability to avoid false-positive warnings, 
and a special (file/macro) malware database was included to determine the 
degree to which trojan horses are detected.

As in test "2001-10", 5 platforms (DOS, W-98, W-NT, W-2000, Linux) were used. 

The databases of macro and script virus and malware were stored on a 
Windows NT 4.0 SP5 server:

    Win-NT Server (1) has the following hardware:
        Pentium 200 MHz, 64 MB RAM, 2 GB hard disk (boot)
                                    2*4,3 GB data/reports, 
                                    2*9,1 GB virus database (mirror)
                   3 network cards: 2*100 MBit/sec, 1*10 MBit/sec
        Protected against electrical faults (USV: APC 420 VA)
        Operating system: Windows NT Server 4.0 SP 6

    Network:       1* 10 MBit/sec BNC for 20 DOS clients
                   1*100 MBit/sec via 2 cascaded switches 
                         for all other clients with 10 MBit/sec cards
                   1*100 MBit/sec via 100 MBit/sec hub for all other clients
                    

Additionally, 25 clients (15 MS-DOS, 9 for Windows platforms: Win-98, Win-NT
and W-2k, and 1 Linux) were used for the test. DOS-Clients work on MS-DOS 6.22. 
Hard disks are only used for the boot process. All W32 client works under 
English version.  Win-NT clients work under Windows NT 4.0 Workstation with SP 5, 
English version. All clients are connected to the server using Microsoft NetBUI.

Generally, clients were flexibly allocated to optimize scanning processes. As the
test is performed in a university lab, with no additional funding from elsewhere
(we also do NOT request AV producers to pay any fee for our tests!), our hardware
may not be regarded "the best possible":

    DOS Clients (15)    have the following hardware:
    ------------------------------------------------
    15* Intel 80486 DX2 50 MHz, 16 MB RAM, 270 MB hard disk, 10 MBit/sec
                    switched to 5 monitors over switchboard
                    software: MS-DOS version 6.22

    Windows Clients (9) have the following hardware:
    ------------------------------------------------
      2*Pentium     133 MHz,     64 MB RAM, 2 GB hard disk,  10 MBit/sec
        Pentium      90 MHz,     32 MB RAM, 1 GB hard disk, 100 MBit/sec
        Pentium-II  350 MHz,     64 MB RAM, 2 GB hard disk, 100 MBit/sec
        Pentium     233 MMX MHz, 64 MB RAM, 2 GB hard disk, 100 MBit/sec
        Pentium-II  233 MHz,     64 MB RAM, 4 GB hard disk, 100 MBit/sec
        Pentium-II  350 MHz,     64 MB RAM, 4 GB hard disk, 100 MBit/sec              
        Pentium MMX 233 MHz     196 MB RAM, 4 GB hard disk, 100 MBit/sec 
        Pentium III             128 MB RAM, 4 GB hard disk, 100 MBit/sec 

    Linux Client (1) has the following hardware:
    --------------------------------------------
        Pentium 166 MHz 64 MB RAM, 100 MBit/Sec
        System: Linux (SuSe) Professional 7.0

    BTW: any donation of related hardware will be warmly welcomed by VTC test team. 


Specially developed software supporting semi-automatic execution of 
test scans and evaluation of protocols consist of batch programs and 
scripts (PERL and AWK). Some UNIX programs like AWK, GAWK, JOIN etc 
have also been applied.


2) The Databases of Macro and Script viruses:
---------------------------------------------
An overview of entries in the VTC virus databases (status: April 30,
2001) is given in Appendix 3: "A3TSTBED.zip" and A4TSTDIR.txt. 
TESTBED.VTC contains the following entries (in ZIPped form):

    1) In-The-Wild Testbeds:
    ------------------------
    ITW-MACR.VTC    content of ITW macro virus testbed
    ITW-SCRI.VTC    content of ITW script virus testbed
    PAC-FILE.VTC    content of packed ITW file virus testbed
    PAC-MACR.VTC    content of packed ITW macro virus testbed
    FP-MACR.VTC     content of Macro virus FalsePositive Testbed

    2) Zoo (=full collection) Testbeds:
    -----------------------------------
    ZOO-MACR.VTC    content of full macro virus testbed
    ZOO-SCRI.VTC    content of full script virus testbed
    MAL-MACR.VTC    content of macro malware testbed
    MAL-SCRI.VTC    content of script malware testbed


These entries (which also indicate the multiplicity of infected 
objects in the resp. directory) also conform with related entries in 
scanner evaluation protocols.

        
The macro virus database is organised according to the CARO macro 
naming convention. Related testbeds contain macro viruses known at 
end-April 2001 (see VTCs List of Known Macro Viruses). For each macro
virus, different goat documents were stored to test consistent 
identification and reliable detection.

    Contents of the macro virus database:
    -------------------------------------
      6,762 different macro viruses
     21,667 files infected each with exactly ONE macro virus
        143 different macro viruses reported "In-The-Wild"
      1,308 files infected with exactly ONE ITW-virus
         80 ITW macro viruses in 672 infected objects, packed 
            with one of 6 packers (ZIP,LHA,ARJ,RAR,WINRAR,CAB)
        329 totally non-malicious/non-viral objects in 26 different
            directories for fp-test

With fast deployment of script (esp. VBS) viruses, a special testbed
for script viruses was developped (the content of which is reflected
in VTCs List of Known Script Viruses).

   Contents of the scriptvirus database:
    ------------------------------------
        588 different script viruses
      1,079 files infected each with exactly ONE script virus
         19 different script viruses reported "In-The-Wild"
        110 files infected with exactly ONE ITW-virus



2B) Additional Macro Malware Database:
--------------------------------------
Concerning non-viral macro malware, this is well documented 
(see VTCs "List of Known Macro Malware" which summarizes both viral 
and non-viral macro malware). This testbed included:

       426 specimen of macro malware in 683 different directories. 


2C) Additional Script Malware Database:
---------------------------------------
Concerning non-viral script malware, this is well documented 
(see VTCs "List of Known Script Malware" which summarizes both viral 
and non-viral script malware). This testbed - which is used for
the first time - included:

       22 specimen of macro malware in 30 different directories. 


2C) Additional test for False Positive Detection:
-------------------------------------------------
In order to test the ability of scanners to avoid "false positive"
alarms on non-malicious non-viral objects (files and macros), 2 sets 
of "clean" objects were mixed into the resp. viral databases.


Clean files collected from several CD-ROMs were used for tests:

      664 non-malicious non-viral objects (*.exe, *.com etc) 
        were stored in 27 different directories.

The list of CD-ROMs used for false positive testing is listed in
appendix 3 (A3TSTBED.ZIP).

Concerning testing for false positive alarms on macro viruses, a set
of
      329 non-malicious non-viral objects (*.doc, *.dot, *.xls)
          were stored in 26 different directories.

Remark: concerning copyright of related CD-ROMS, we use selected active
content to help protecting the copyright holder for wrong allegations
concerning false alarms. We never use the code actively but only for
assurance that scanners dont falsely alarm on these samples. 


6.) Testing scanners on standard database of Macro Viruses:
-----------------------------------------------------------
All AV scanners are tested against two large macro-related 
database. The main datadabse contains all "zoo" and ITW macro 
viruses, both in uncompressed and compressed forms; mixed into this
dfatabase, there are also specific directories contaiining non-viral
macro objects for false-positive detection. The second (smaller)
database contains all non-viral maco malware (trojans, droppers, 
intendeds etc). All malware included in those databases matches 
the contents of the VTC Macro Virus List, which is published  
regularly (previously: monthly, now at the end of each quarter)
For details, see http://agn-www.informatik.uni-hamburg.de/vtc. 

The malware database contains also some file viruses which are being 
created ("dropped") by macro viruses. We decided to test them in the 
context of the macro malware test because they only appear in the 
context of macro malware.

The directory structure of the virus database reflects the CARO 
naming scheme for macro viruses with all samples of one variant 
stored in one subdirectory. Starting from the root directory of the
database, the first level contains directories describing the host 
software (Word, Word97, Excel, Excel97, Lotus123, AmiPro). The second
level contains subdirectories with the names of the families of the 
viruses and the next level hosts subdirectories of all variants of 
that family, in which the viruses can be found. Optionally (only in
malware database), we have another subdirectory called "FILE" which 
contains the file viruses mentioned above.  

The number of samples for each virus varies between one and 78 
samples (for Concept.A), although the average is 2-3 infected objects
each. Our results are split into two sections: "detection of viruses"
and "detection of files", where "detection of viruses" has two sub-
sections: "unreliable detection" and "unreliable identification".

(An index of the malware databases is available in a3tstbed.zip)

After each scanner is run, all report files are preprocessed by those
AWK scripts already mentioned in the desciption of file virus test.


7.) Testing scanners on standard database of Script Viruses:
------------------------------------------------------------
The test is equivalent to th macro virus test except that the testbed
is bases on script viruses the status of which is regularly published
by VTC in the "List of Known Script Malware" (LoKSM) (see VTC website).
Presently, the script virus testbed adresses the following platforms:

	VBS, JS, IRC, mIRC et al.


8.) Creating the final summary of the results:
----------------------------------------------
(Text essentially same as in previous test: 2000-08 / 2001-04).

The final evaluations for all tests are similar. Only one report of 
file and macro viruses tests is used to get the total number of files 
in the directory. As for boot viruses, the configuration file from 
Simboot is used (if there was no specific need for manual operation). 
Three new files result from these processes. New files contain the 
directory name and the total number of files in this directory. Each 
preprocessed report is joined with the new file. One AWK-scripts 
evaluates the result of the joining.

The results are listed as follows:
   - The number of viruses (+malware) detected: it is not necessary 
     that all examples of the virus are detected.
   - The number of viruses with unreliable (=inconsistent) 
     identification: all variants of a viruses are detected 
     but at least one sample is identified with a different name.
   - The number of viruses with unreliable detection: here, not all 
     samples of a virus are detected but at least one.

The files containing the preprocessed information mentioned above are
huge, although they are reduced to contain essentially the virus names. 
For all tested scanners (latest version), they are included in a 
separate archive (Scan-Res) for anonymous ftp. 

