=========================================
            File 7EVAL-WNT.TXT
=========================================
   Evaluation of results for Macro and 
     Script Virus/Malware detection 
   under Windows-NT in Heureka-2 test
=========================================
Formatted with non-proportional font (Courier)

Forword: This test is based on VTC test "2001-09". Products submitted for
         that test (with engines and signatures dated before June 25, 2001)
         were applied to 2 differential testbeds:

	   Testbed ".017" contained all those macro/script viruses/malware
                          which were reported between May 1, 2001 and
                          July 31, 2001. 
           Testbed ".01A" contained all those macro/script viruses/malware
                          which were reported between August 1 and 
            		  October 31, 2001.
           For both testbeds, only those viruses were icluded in In-The-Wild
                          tests which have been newly reported as "In-The-
                          Wild" during the related period.

	 The goal of test "2002-02" is to determine to which degree AV products
         are able to reliably detect macro/script viruses and malware found
         after delivery of engines and signatures. In order to enable compa-
         rison with VTC test "2001-09", products were used in the same mode,
         esp. with the same options and parameters as in that test. Therefore,
         this is NOT EXACTLY a test for heuristic detection (where one should
         set switches/options accordingly, e.g. to exclude detection by signa-
         tures). ("Heureka" is a word from ancient Greek meaning "I found it").
          

  Content of this file:
  ************************************************************************
    Eval WNT.MZ: Development of Macro Zoo (MZ) virus detection rates
                    Table WNT.MZ: Development of MZ detection rates
                    Analysis of Macro Zoo virus detection rates
		    Result Heureka-2.MZ (Macro Zoo viruses)
    Eval WNT.MI: Development of Macro ITW (MI) virus detection rates
                    Table WNT.MI: Development of MI detection rates
                    Analysis of Macro ITW virus detection rates
		    Result Heureka-2.MI (Macro ITW viruses)
    Eval WNT.MM: Development of Macro Malware (MM) detection rates
                    Table WNT.MM:Development of MM detection rates
                    Analysis of Macro Malware detection rates
		    Result Heureka-2.MM (Macro Malware)
    Eval WNT.SZ: Development of Script Zoo (SZ) virus detection rates
                    Table WNT.SZ: Development of SZ detection rates
                    Analysis of Script Zoo virus detection rates
		    Result Heureka-2.SZ (Script Zoo viruses)
    Eval WNT.SI: Development of Script ITW (SI) virus detection rates
                    Table WNT.SI: Development of SI detection rates
                    Analysis of Script ITW virus detection rates
		    Result Heureka-2.SI (Script ITW viruses)
    Eval WNT.SM: Development of Script Malware (SM)  detection rates
                    Table WNT.SM: Development of SM detection rates
                    Analysis of Script Malware detection rates
		    Result Heureka-2.SM (Script Malware)

    COMPARE:     Comparison of Heureka-1 and Heureka-2 test results

    Eval WNT.SUM Grading of WNT products according to "Heureka-2" results
  ************************************************************************


This part of VTC "2002-02" test report evaluates the detailed results as 
given in section (file):

           6GWNT.TXT       Macro/Script Viruses/Malware results (W-NT)


The following (21) products participated in this special "heuristic" 
scanner test for WNT products:

	 --------------------------------------------------------
           Products submitted for aVTC test under Windows-NT:
	 --------------------------------------------------------
     	   ANT     v(def): 6.8.0.2 	    sig: June 05,2001
	   AVA     v(def): unknown	    sig: unknown
 	   AVG     v(def): 6.0.263	    sig: June 22,2001
 	   AVK     v(def): 10.0.167	    sig: June 21,2001
 	   AVP     v(def): 3.5.133.0	    sig: June 01,2001
 	   AVX     v(def): 6.1              sig: June 18,2001
 	   CMD     v(def): 4.61.5           sig: June 25,2001
 	   DRW     v(def): 4.25             sig: June 20,2001
 	   FPR     v(def): 3.09d	    sig: June 25,2001
 	   FPW     v(def): 3.09d	    sig: June 25,2001
 	   FSE     v(def): 1.00.1251        sig: June 20,2001
		           scan eng fprot:  3.09.507
		    	   scan eng avp:    3.55.3210
		           scan eng orion:  1.02.15
 	   IKA     v(def): 5.01	            sig: June 25,2001
 	   INO     v(def): 6.0.85  	    sig: June 14,2001
	   MR2     v(def): 1.17    	    sig: June 25,2001
 	   NVC     v(def): 5.00.25 	    sig: June 19,2001
 	   PAV     v(def): 3.5.133.0	    sig: June 23,2001
 	   QHL     v(def): 6.02    	    sig: June 28,2001
 	   RAD     v(def): 8.1.001	    sig: June 25,2001
 	   RAV     v(def): 8.2.001,
                    	   scan eng:8.3     sig: June 25,2001
 	   SCN     v(def): 4144
                    	   scan eng:4.1.40  sig: June 20,2001
 	   VSP     v(def): 12.22.1 	    sig: June 25,2001
         --------------------------------------------------------
	 One products (NAV) was withdrawn from this test, 
         due to "new engines".


Eval WNT.MZ: Development of Macro Zoo virus detection rates:
============================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | Loss in |   New viruses | Loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001 I   July 31,2001I         IOctober 31,2001I
Testbed    6762 100.0%  |    357 100.0% I         |    176 100.0% I
------------------------+---------------+---------+---------------+----------
ANT        6566  97.1%  |    221  61.9% |  -35.2% |     77  43.8% |  -53.3%
AVA        6604  97.7%  |    254  71.1% |  -26.6% |     97  55.1% |  -42.6%
AVG        6651  98.4%  |    318  89.1% |   -9.3% |    117  66.5% |  -31.9%
AVK        6762 100.0%  |    288  80.7% |  -19.3% |     69  39.2% |  -60.8%
AVP        6761 100.0%  |    292  81.8% |  -18.2% |     70  39.8% |  -60.2%
AVX        6703  99.1%  |    343  96.1% |   -3.0% |    166  94.3% |   -4.8%
CMD        6760 100.0%  |    324  90.8% |   -9.2% |    128  72.7% |  -27.3%
DRW        6725  99.5%  |    344  96.4% |   -3.1% |    169  96.0% |   -3.5%
FPR        6760 100.0%  |    322  90.2% |   -9.8% |    127  72.2% |  -27.8%
FPW        6760 100.0%  |    322  90.2% |   -9.8% |    127  72.2% |  -27.8%
FSE        6762 100.0%  |    341  95.5% |   -4.5% |    151  85.8% |  -14.2%
IKA        6451  95.4%  |    290  81.2% |  -14.2% |    107  60.8% |  -34.6%
INO        6755  99.9%  |    339  95.0% |   -4.9% |    167  94.9% |   -5.0%
MR2          44   0.7%  |      6   1.7% |    1.0% |      5   2.8% |    2.1%
NVC        6751  99.8%  |    223  62.5% |  -37.3% |     50  28.4% |  -71.4%
PAV        6762 100.0%  |    292  81.8% |  -18.2% |     70  39.8% |  -60.2%
QHL           0   0.0%  |      0   0.0% |    0.0% |      0   0.0% |    0.0%
RAV        6726  99.5%  |    330  92.4% |   -7.1% |    134  76.1% |  -23.4%
SCN        6762 100.0%  |    349  97.8% |   -2.2% |    167  94.9% |   -5.1%
VSP           1   0.0%  |      0   0.0% |    0.0% |      1   0.6% |    0.6%
------------------------+---------------+---------+---------------+----------
Mean ALL:        85.7%		  72.8%    -11.5%	    56.8%    -27.6%
Mean rel:	 99.2%		  85.6%    -10.5%	    66.6%    -25.1%
------------------------+---------------+---------+---------------+----------
Remarks: "Mean ALL" is the mean value of virus and file identification 
            calculated over ALL related entries.
         "Mean rel" is the relative mean value of virus and file 
            identification calculated only for those entries
	    with a minimum detection rate "minrate", where
		   minrate = 65% for zoo viruses, 
                   minrate = 95% for ITW viruses, and
                   minrate = 60% for malware.

         Definition of "loss vector":
	    Loss in 3 months = Loss vector #1 
                             = detection rate in month 1-3
                               minus detection rate in reference test
            Loss in 6 months = Loss vector #2 
                             = detection rate in month 4-6
                               minus detection rate in reference test
 

Analysis of results for Macro Zoo viruses:
==========================================
	  (1) For zoo macro viruses, best products are able to 
              detect more than 90% of those viruses reported within 
              first 3-month period and more than 80% within second 
              3-month period after product/signature delivery:

			SCN   (100.0%  97.8%  94.9%)
			FSE   (100.0%  95.5%  85.8%)
		        INO   ( 99.9%  95.0%  94.9%)
			DRW   ( 99.5%  96.4%  96.0%)
			AVX   ( 99.1%  96.1%  96.0%)
	      
	  (2) During the first 3 months, mean loss in detection
              ability is 11.5% (overall), and it is slightly
              better (10.5%) when products with extremely low 
	      detection rates are not counted. The following
              products behave best in first 3-month period:

			SCN        (97.8%)
                        DRW        (96.4%)
                        AVX        (96.1%)
                        FSE        (95.5%)
                        INO        (95.0%)

	  (3) In months 4-6, the loss in detection quality
              is fastly growing, with a mean loss of 27.6%
              (overall) and 25.1% when products with extremely 
              low detection rates are not counted.

	      For viruses first reported in the next 3 months
              (=fourth to sixt month after product delivery), 
              best products still detect more than 90% of
              zoo macro viruses:

			DRW        (96.0%)
			INO        (94.9%)
                        SCN        (94.9%)
			AVX        (94.3%)  

	  (4) In order to classify product behaviour, we grade 
              products according to loss in detection quality. 
              When considering only products with losses up to 20% 
              after 6 months (ordered according to highest 
	      detection rates after 6 months), the following 
              products behaved best in "Heureka-2" test:

                        ------------------------------------
			detection rate   loss in    loss in 
           AV product    in ref-test    month 1-3  month 4-6
			------------------------------------
	      DRW	      99.5%      -3.1%       -3.5%
	      SCN	     100.0%	 -2.2%       -5.1%
	      INO             99.9%      -4.9%       -5.0%   
	      AVX             99.1%      -3.0%       -4.8%
	      FSE            100.0%      -4.5%      -14.2%
			------------------------------------


     *************************************************************
     Result "Heureka-2.MZ":  concerning new zoo macro viruses,
                             the following 4 products miss 
                             less than 10% over 6 months:
              -----------------------------------------------------
		DRW     after 3 months:  -  3.1% 	
                        after 6 months:  -  3.5%
              -----------------------------------------------------
		SCN     after 3 months:  -  2.2% 	
                        after 6 months:  -  5.1%
              -----------------------------------------------------
		INO     after 3 months:  -  4.9% 	
                        after 6 months:  -  5.0%
              -----------------------------------------------------
		AVX     after 3 months:  -  3.0% 	
                        after 6 months:  -  4.8%
     **************************************************************
             And the following product misses less than 20% 
             over 6 months:
		FSE     after 3 months:  - 4.5% 	
                        after 6 months:  -14.2%
     **************************************************************


Eval WNT.MI: Development of Macro ITW virus detection rates:
============================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | loss in |   New viruses | loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001     July 31,2001          IOctober 31,2001 
Testbed     143 100.0%  |     17 100.0%           |      7 100.0%
------------------------+---------------+---------+---------------+----------
ANT         142  99.3%  |     14  82.4% |  -16.9% |      5  71.4% |  -27.9%
AVA         143 100.0%  |     16  94.1% |   -5.9% |      5  71.4% |  -28.6%
AVG         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
AVK         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
AVP         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
AVX         143 100.0%  |     16  94.1% |   -5.9% |      6  85.7% |  -14.3%
CMD         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
DRW         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
FPR         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
FPW         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
FSE         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
IKA         142  99.3%  |     17 100.0% |    0.7% |      7 100.0% |    0.7%
INO         143 100.0%  |     16  94.1% |   -5.9% |      7 100.0% |    0.0%
MR2          13   9.1%  |      0   0.0% |   -9.1% |      0   0.0% |   -9.1%
NVC         143 100.0%  |     17 100.0% |    0.0% |      6  85.7% |  -14.3%
PAV         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
QHL           0   0.0%  |      0   0.0% |    0.0% |      0   0.0% |    0.0%
RAV         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
SCN         143 100.0%  |     17 100.0% |    0.0% |      7 100.0% |    0.0%
VSP           0   0.0%  |      0   0.0% |    0.0% |      0   0.0% |    0.0%
------------------------+---------------+---------+---------------+----------
Mean ALL:	 86.7%		  86.7%	    -2.2%	    80.7%     -4.7%	        
Mean rel:	 99.9%		  99.9%	    -2.4%	    95.0%     -5.2%
------------------------+---------------+---------+---------------+----------
Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ"


Analysis of results for Macro ITW viruses:
==========================================
	  (0) Due to the small number of ITW Macro viruses detected in 
              each 3-month period, we just discuss findings but dont grade
              products based on such potentially insignificant figures.

	  (1) For macro ITW viruses, the majority of products detect
 	      all ITW viruses even after 6 months. The following products
              consistently detect ALL macro ITW viruses at reference test
	      as well after 3 and 6 months, ALL with perfect detection 
	      vectors (100% 100% 100%):

	               AVG, AVK, AVP, CMD, DRW, FPR, FPW, 
                            FSE, INO, PAV, RAV, SCN.

	   (2) In comparison with Heureka-1 test, where 6 products
               detected ALL ITW macro viruses, the situation has
               improved significantly.             


     *************************************************************
     Result "Heureka-2.MI":  concerning new Macro ITW viruses,
                             the following 13 products miss
                             NO ITW virus during 6months:
	                     AVG, AVK, AVP, CMD, DRW, FPR, FPW, 
                             FSE, INO, PAV, RAV, SCN.
     **************************************************************


Eval WNT.MM: Development of macro malware detection rates:
==========================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | loss in |   New viruses | loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001 I   July 31,2001I         IOctober 31,2001I
Testbed     426 100.0%  |     22 100.0%           |      7 100.0%
------------------------+---------------+---------+---------------+----------
ANT         378  88.7%  |     10  45.5% |  -43.2% |      4  57.1% |  -31.6%
AVA         377  88.5%  |     11  50.0% |  -38.5% |      3  42.9% |  -45.6%
AVG         352  82.6%  |     14  63.6% |  -19.0% |      1  14.3% |  -68.3%
AVK         425  99.8%  |     12  54.5% |  -45.3% |      3  42.9% |  -56.9%
AVP         425  99.8%  |     14  63.6% |  -36.2% |      3  42.9% |  -56.9%
AVX         392  92.0%  |     18  81.8% |  -10.2% |      7 100.0% |    8.0%
CMD         424  99.5%  |     15  68.2% |  -31.3% |      4  57.1% |  -42.4%
DRW         387  90.8%  |     18  81.8% |   -9.0% |      7 100.0% |    9.2%
FPR         424  99.5%  |     15  68.2% |  -31.3% |      4  57.1% |  -42.4%
FPW         424  99.5%  |     15  68.2% |  -31.3% |      4  57.1% |  -42.4%
FSE         425  99.8%  |     17  77.3% |  -22.5% |      4  57.1% |  -42.7%
IKA         383  89.9%  |     16  72.7% |  -17.2% |      5  71.4% |  -18.5%
INO         398  93.4%  |     15  68.2% |  -25.2% |      5  71.4% |  -22.0%
MR2         135  31.7%  |      0   0.0% |  -31.7% |      2  28.6% |   -3.1%
NVC         421  98.8%  |     12  54.5% |  -44.3% |      3  42.9% |  -55.9%
PAV         426 100.0%  |     14  63.6% |  -36.4% |      3  42.9% |  -57.1%
QHL           0   0.0%  |      0   0.0% |    0.0% |      0   0.0% |    0.0%
RAV         416  97.7%  |     17  77.3% |  -20.4% |      5  71.4% |  -26.3%
SCN         426 100.0%  |     17  77.3% |  -22.7% |      3  42.9% |  -57.1%
VSP           1   0.2%  |      0   0.0% |   -0.2% |      0   0.0% |   -0.2%
------------------------+---------------+---------+---------------+----------
Mean ALL:	 83.4% 		  56.8	   -25.8%	    50.0%    -32.6%
Mean rel:	 95.0%		  66.8%	   -28.7%           55.6%    -36.2%
------------------------+---------------+---------+---------------+----------
Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ"


Analysis of results for Macro Malware:
======================================
	  (0) Due to the small number of Macro Malware detected in each
              3-month period, we just discuss findings but dont grade
              products based on such potentially insignificant figures.

	  (1) For non-replicant Macro Malware, detection quality is - in the 
              mean - significantly less developed than the detction of 
              replicative malware (aka viruses & worms). The mean malware 
              detection rate of tested products (except those with extremely 
              insufficient detection rates) degrades from 95.0% (in reference 
              test) to 66.8% (after 3 months) further down to 55.6% (after
              6 months). 

	  (2) Some products even improve their detection rates, as the
 	      following detection vectors indicate:

			DRW ( 90.8%   81.8%  100.0%)
			AVX ( 92.0%   81.8%  100.0%)

	      This may indicate that the heuristic mechanism of these 
              products are very well developed, but with the relative 
              small set of samples (7 for months 4-6), it canNOT be
              determined whether this results is an artefact of the 
              statistical evaluation.

	  (3) The following products loose less than 20% detection rate
              over each 3-month period but they they dont start with
	      optimum detection rate in the reference test:

			DRW ( 90.8%   81.8%  100.0%)
			AVX ( 92.0%   81.8%  100.0%)
			IKA ( 89.9%   72.7%   71.4%)

	  (4) Those products which detected almost all malware samples
              with "fresh" signatures in the reference test (esp. PAV
              and SCN) lost significantly more detection rate compared 
              to mean loss. This may indicate that these products apply
              mechanisms of exact identification instead of heuristics.

	  (5) In comparison with Heureka-1 test results, those products
	      then scoring best (FSE, SCN: loss after 6 months: -31.7%)
              have now much larger loss in detection rate (-42.7%, -57.1%).
             

     *******************************************************************
     Result "Heureka-2.MM": The persistency of non-replicative malware
                         detection needs significant improvement. Only
			 3 products loose less than 40% detection 
                         quality over six months, but all three products 
                         have less than optimum detection rates in the
                         reference test.
      *******************************************************************


Eval WNT.SZ: Development of Script Zoo virus detection rates:
=============================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | loss in |   New viruses | loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001 I   July 31,2001I         IOctober 31,2001I
Testbed     588 100.0%  |    164 100.0%           |    102 100.0%
------------------------+---------------+---------+---------------+----------
ANT         481  81.8%  |     42  25.6% |  -56.2% |     12  11.8% |  -70.0%
AVA         174  29.6%  |     32  19.5% |  -10.1% |     11  10.8% |  -18.8%
AVG         370  62.9%  |     85  51.8% |  -11.1% |     40  39.2% |  -23.7%
AVK         588 100.0%  |    126  76.8% |  -23.2% |     52  51.0% |  -49.0%
AVP         588 100.0%  |    126  76.8% |  -23.2% |     49  48.0% |  -52.0%
AVX         412  70.1%  |     89  54.3% |  -15.8% |     31  30.4% |  -39.7%
CMD         552  93.9%  |    104  63.4% |  -30.5% |     46  45.1% |  -48.8%
DRW         561  95.4%  |    136  82.9% |  -12.5% |     72  70.6% |  -24.8%
FPR         558  94.9%  |    104  63.4% |  -31.5% |     46  45.1% |  -49.8%
FPW         556  94.6%  |    104  63.4% |  -31.2% |     46  45.1% |  -49.5%
FSE         588 100.0%  |    141  86.0% |  -14.0% |     71  69.6% |  -30.4%
IKA         457  77.7%  |    104  63.4% |  -14.3% |     49  48.0% |  -29.7%
INO         559  95.1%  |     78  47.6% |  -47.5% |     34  33.3% |  -61.8%
MR2         490  83.3%  |     93  56.7% |  -26.6% |     45  44.1% |  -39.2%
NVC         537  91.3%  |     74  45.1% |  -46.2% |     27  26.5% |  -64.8%
PAV         588 100.0%  |    126  76.8% |  -23.2% |     47  46.1% |  -53.9%
QHL           1   0.2%  |      1   0.6% |    0.4% |      1   1.0% |    0.8%
RAV         485  82.5%  |      0   0.0% |  -82.5% |      0   0.0% |  -82.5%
SCN         587  99.8%  |    134  81.7% |  -18.1% |     58  56.9% |  -42.9%
VSP         494  84.0%  |     93  56.7% |  -27.3% |     45  44.1% |  -39.9%
------------------------+---------------+---------+---------------+----------
Mean ALL:	 78.7%		  54.6%	   -27.2%	    38.3%    -43.6%
Mean rel:        86.6%		  60.7%	   -28.6%	    42.5%    -45.8%
------------------------+---------------+---------+---------------+----------
Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ"


Analysis of results for Script Zoo viruses:
===========================================
	  (1) Heuristic detection of script viruses is significantly
              less developed than detection fo macro viruses, as comparison
              of mean losses (without those products with inadequate 
              detection rates) in detection rates shows:

						reference   after    after
	                                          test    3 months  6 months
	      detection rate of macro viruses     99.2%     85.6%     66.6%
	      detection rate of script viruses    86.6%     60.7%     42.5%

	      For zoo script viruses, best products are able to detect 
              more than 80% of those zoo viruses reported within 3-months 
              after products delivery and 60% of those viruses reported
              after 6 months:
			
			FSE   (100.0%  86.0%  69.9%)
			DRW   ( 95.4%  82.9%  70.6%)

	      In addition, the following products (which detected at least 
              90% in the reference test) lost less than 20% in the first 
              3-month period but lost more than 40% in the second period:

			SCN   ( 99.8%  81.7%  56.9%)

      	  (2) During the first 3 months, mean loss in detection
              ability is 28.6%. In months 4-6, the loss in detection 
              quality is fastly growing, with a mean loss of 45.8%.
           
          (3) In order to classify product behaviour, we grade 
              products according to loss in detection quality. 
              When considering only products with losses up to 
              40% after 6 months (ordered according to highest 
              detection rates after 6 months), the following 
              products behaved best in "Heureka-2" test:

                        ------------------------------------
			detection rate   loss in    loss in 
           AV product    in ref-test    month 1-3  month 4-6
	      --------+-------------------------------------
	      DRW	      95.4%     -12.5%      -24.8%
	      FSE            100.0%     -14.0%      -30.4%
                        ------------------------------------
	      SCN	      99.8%	-18.1%      -42.9%
			------------------------------------


     ********************************************************************
     Result "Heureka-2.SZ": Zoo script virus detection is significantly
                            less well developped compared with macro virus
                            detection; losses in detection rates are more
                            than 5 times higher than with macro viruses. 

			    The following 2 products miss less than 15%
                            after 3 months and about 30% after 6 months:

				DRW     after 3 months:  - 12.5% 	
                        		after 6 months:  - 24.8%
                                --------------------------------
			 	FSE     after 3 months:  - 14.0% 	
                        		after 6 months:  - 30.4%

                            For AV companies, there is strong need to
			    improve persistent detection methods esp. as 
                            this category adresses many mass-emailing 
                            viruses!

			    For customers, the strong evidence is to
                            update AV products for script virus detection
                            much more often than for script viruses. 
     ********************************************************************



Eval WNT.SI: Development of Script ITW virus detection rates:
=============================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | loss in |   New viruses | loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001 I   July 31,2001I         IOctober 31,2001I
Testbed      19 100.0%  |     10 100.0%           |      6 100.0%
------------------------+---------------+---------+---------------+----------
ANT          19 100.0%  |     10 100.0% |    0.0% |      2  33.3% |  -66.7%
AVA          18  94.7%  |     10 100.0% |    5.3% |      2  33.3% |  -61.4%
AVG          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
AVK          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
AVP          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
AVX          19 100.0%  |     10 100.0% |    0.0% |      2  33.3% |  -66.7%
CMD          19 100.0%  |     10 100.0% |    0.0% |      5  83.3% |  -16.7%
DRW          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
FPR          19 100.0%  |     10 100.0% |    0.0% |      5  83.3% |  -16.7%
FPW          19 100.0%  |     10 100.0% |    0.0% |      5  83.3% |  -16.7%
FSE          19 100.0%  |     10 100.0% |    0.0% |      5  83.3% |  -16.7%
IKA          18  94.7%  |     10 100.0% |    5.3% |      4  66.7% |  -28.0%
INO          19 100.0%  |      7  70.0% |  -30.0% |      1  16.7% |  -83.3%
MR2          17  89.5%  |     10 100.0% |   10.5% |      1  16.7% |  -72.8%
NVC          19 100.0%  |     10 100.0% |    0.0% |      3  50.0% |  -50.0%
PAV          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
QHL           1   5.3%  |      1  10.0% |    4.7% |      1  16.7% |   11.4%
RAV          18  94.7%  |      0   0.0% |  -94.7% |      0   0.0% |  -94.7%
SCN          19 100.0%  |     10 100.0% |    0.0% |      4  66.7% |  -33.3%
VSP          17  89.5%  |     10 100.0% |   10.5% |      1  16.7% |  -72.8%
------------------------+---------------+---------+---------------+----------
Mean ALL:	 89.5%		  89.0%	    -4.4%	    50.8%    -43.7%           
Mean rel:	 98.2%		  98.3%	    -4.4%	    53.5%    -45.4%   			          
------------------------+---------------+---------+---------------+----------
Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ"


Analysis of results for Script ITW viruses:
===========================================
	  (0) Due to the small number of ITW Script viruses detected in 
              each 3-month period, we just discuss findings but dont grade
              products based on such potentially insignificant figures.


	  (1) For script ITW viruses, the majority of products detect
 	      all ITW viruses after 3 months but detection rates are signi-
              ficantly reduced after 6 months. 


     **********************************************************
     Result "Heureka-2.SI": concerning new script ITW viruses,
	                    detection rates degrade much faster
                            after 3 months than for macro ITW 
                            viruses. 
     **********************************************************


Eval WNT.SM: Development of Script Malware detection rates:
============================================================
------------------------+---------------+---------+---------------+----------
             Viruses    |   New viruses | loss in |   New viruses | loss in 
Scanner      detected   |    detected   | 3 months|    detected   | 6 months
------------------------+---------------+---------+---------------+----------
Status:   April 30,2001 I   July 31,2001I         IOctober 31,2001I
Testbed       22 100.0  |     37 100.0%           |     73 100.0%
------------------------+---------------+---------+---------------+----------
ANT           0   0.0%  |      7  18.9% |   18.9% |      5   6.8% |    6.8%
AVA         ---     0%  |      1   2.7% |    2.7% |      2   2.7% |    2.7%
AVG           5  22.7%  |      4  10.8% |  -11.9% |      4   5.5% |  -17.2%
AVK          22 100.0%  |     25  67.6% |  -32.4% |     20  27.4% |  -72.6%
AVP          22 100.0%  |     28  75.7% |  -24.3% |     20  27.4% |  -72.6%
AVX           2   9.1%  |     10  27.0% |   17.9% |      8  11.0% |    1.9%
CMD          14  63.6%  |      8  21.6% |  -42.0% |      4   5.5% |  -58.1%
DRW           8  36.4%  |     19  51.4% |   15.0% |     21  28.8% |   -7.6%
FPR          14  63.6%  |      8  21.6% |  -42.0% |      4   5.5% |  -58.1%
FPW          14  63.6%  |      8  21.6% |  -42.0% |      4   5.5% |  -58.1%
FSE          22 100.0%  |     32  86.5% |  -13.5% |     24  32.9% |  -67.1%
IKA           8  36.4%  |     15  40.5% |    4.1% |     11  15.1% |  -21.3%
INO          15  68.2%  |      9  24.3% |  -43.9% |     14  19.2% |  -49.0%
MR2           5  22.7%  |      4  10.8% |  -11.9% |     10  13.7% |   -9.0%
NVC           2   9.1%  |      5  13.5% |    4.4% |      4   5.5% |   -3.6%
PAV          22 100.0%  |     25  67.6% |  -32.4% |     20  27.4% |  -72.6%
QHL           1   4.5%  |      1   2.7% |   -1.8% |      1   1.4% |   -3.1%
RAV          18  81.8%  |      0   0.0% |  -81.8% |      0   0.0% |  -81.8%
SCN          22 100.0%  |     27  73.0% |  -27.0% |     21  28.8% |  -71.2%
VSP           5  22.7%  |      4  10.8% |  -11.9% |     10  13.7% |   -9.0%
------------------------+---------------+---------+---------------+----------
Mean ALL:        49.6%		  32.4%	   -32.4%	    14.2%    -36.0%
Mean rel:	(63.1%)		 (37.8%)  (-46.0%)         (22.3%)  (-68.2%)
------------------------+---------------+---------+---------------+----------
Remark: concerning calculation of mean values: see 1st table "Eval WNT.MZ"


Analysis of results for Script Malware:
=======================================
	  (0) When comparing the numbers of macro and script malware detected 
              within two consecutiv 3-month periods, much more samples of
              script malware have been detected for the latter.

	  (1) For non-replicant Script Malware, detection quality starts
              at a significantly lower level (63.1%), and quality degrades
              much faster than for replicative malware (aka viruses & worms). 
              The mean malware detection rate of tested products (except 
              those with extremely insufficient detection rates) degrades 
              from 63.1% (in reference test) to 37.8% (after 3 months) 
              further down to 22.3% (after 6 months). 
             

     *******************************************************************
     Result "Heureka-2.SM": The persistency of non-replicative malware
                         detection needs significant improvement. 
                         Customers of AV products are advised to update
                         there products much faster for detection of
                         trojanic script malware than for macro malware.
      *******************************************************************



Comparing results of test Heureka-1 and Heureka-2:
===================================================
The following table lists essential results of Heureka tests:

				    Mean detection rates in:
			       Heureka-1		   Heureka-2
                      ---------------------------+-----------------------------
				after     after                after     after
		    reference  3 months  6 months  reference  3 months  6 months
-------------------------------------------------+-----------------------------
MZ=Macro zoo viruses   90.8%     73.7%     66.0%     99.2%	85.6%	  66.6%
MI=Macro ITW viruses   91.8%     89.7%     83.0%     99.9%      99.9%     95.0%
MM=Macro Malware       87.1%     61.8%     56.4%     95.0%      66.8%     55.6%
--------------------------------------------------------------------------------
SZ=Script zoo viruses  83.4%     61.0%     49.3%     86.6%      60.7%     42.5%
SI=Script ITW viruses   ---	  ---	    ---      98.2%      98.3%     53.5%
SM=Script Malware       ---	  ---	    ---      63.1%      37.8%     22.3%
--------------------------------------------------------------------------------

Concerning detection rates for macro viruses, both In-The-Wild and in zoo,
AV products have improved their detection rates both generally and after
first 3-month period significantly. But loss of detection quality in the
second 3-month period is much stronger than before (the results for ITW 
viruses may be influenced to the small number of newly found viruses).

Concerning zoo script viruses, detection rates are stable on an insufficient
level.



Eval WNT.SUM Grading of WNT products according to "Heureka-2" results
=====================================================================
In comparing products, some behave "rather well". Over two 3-months periods,
the following products behaves best (although they also need significant 
improvement):
				DRW and FSE

Moreover, the following products behave best in the first 3-month period:

			 DRW, FSE, SCN and AVX, INO.
	
But as some testbeds were rather small, and as the loss in script viruses
detection quality was so dominant, we decided NOT to grade any product 
in VTCs grading scheme. We nevertheless hope that AV companies do their 
best to improve the generic and heuristic detection mechanisms. And we
strongly advise customers, to upgrade their products and signatures as 
often as possible.


