{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df031ffc",
   "metadata": {},
   "source": [
    "# Step 0 Cluster the Malware Bazaar data set\n",
    "\n",
    "This is in the file (which has been checked in):\n",
    "    malbaz/cen_389300.csv\n",
    "\n",
    "On the 2021-09-17 the CSV file provided by Malware Bazaar had 389300 lines.\n",
    "To download an updated version and cluster this data set, see malbaz/README.\n",
    "Using HAC-T with a threshold distance CDist=30, the resulting clustering had 16453 clutsters.\n",
    "\n",
    "The clusters are described in the file malbaz/cen_389300.csv, which has the following columns\n",
    "    tlsh       TLSH of the center of the cluster\n",
    "    family     The most common \"signature\" in the cluster\n",
    "    firstSeen  The first seen date for the cluster (earliest of all first seen dates)\n",
    "    label      The following concatenated: family, firstseen, nitems\n",
    "    radius     The radius of the cluster\n",
    "    nitems     The number of items in the cluster\n",
    "\n",
    "Below we list the 20 most frequently occurring family assigned to clusters:\n",
    "\n",
    "3416    AgentTesla \\\n",
    "2782    NULL \\\n",
    "942     Heodo \\\n",
    "725     AveMariaRAT \\\n",
    "721     Mirai \\\n",
    "708     FormBook \\\n",
    "580     Loki \\\n",
    "558     QuakBot \\\n",
    "416     RemcosRAT \\\n",
    "382     Dridex \\\n",
    "316     NanoCore \\\n",
    "311     IcedID \\\n",
    "304     SnakeKeylogger \\\n",
    "244     TrickBot \\\n",
    "242     GuLoader \\\n",
    "231     Gozi \\\n",
    "214     RedLineStealer \\\n",
    "210     CobaltStrike \\\n",
    "197     MassLogger \\\n",
    "184     njrat\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9cee6ab",
   "metadata": {},
   "source": [
    "# Step 1 Display a dendogram for all clusters assigned to a malware family\n",
    "For our first demonstration,\n",
    "we selected FickerStealer because the dendrogram was not too large / too dense.\n",
    "Note: We provide some tools for narrowing the search / showing more meaningful dendrograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9fa2371f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 43360 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\", searchColName=\"family\", searchValueList=['FickerStealer'])\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91a3872a",
   "metadata": {},
   "source": [
    "## Interpretation of FickerStealer dendrogram\n",
    "\n",
    "We see a set of close clusters (distances between clusters < 110) in the months of March and April.\n",
    "We see a set of even closer clusters (distances between clusters < 60) in the months of May and August.\n",
    "There may have been a significant change in the malware family between April and May 2021."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90bb2c14",
   "metadata": {},
   "source": [
    "# Step 2 Show a dendrogram for the RacoonStealer family\n",
    "# Use the date filtering options (sDate and eDate)\n",
    "\n",
    "We start by generating a dendrogram for the entire RacoonStealer malware family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f0ab5520",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "warning: The list of TLSH values is too long to show a sensible dendrogram.\n",
      "It is recommended that you filter to a smaller list of TLSH values.\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 55516 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\", searchColName=\"family\", searchValueList=['RaccoonStealer'])\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38093dc1",
   "metadata": {},
   "source": [
    "## 2.1 Use the sDate parameter to specify clusters after a date\n",
    "\n",
    "The above dendrogram was not useful.\n",
    "So we set the start date (sDate) paremeter to \"2021-09-01\", so that we only show clusters with a firstSeen date\n",
    "which occurs on or after 2021-09-01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "47fd669f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 56340 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\", searchColName=\"family\", searchValueList=['RaccoonStealer'],\n",
    "                              sDate=\"2021-09-01\")\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57299c8f",
   "metadata": {},
   "source": [
    "## 2.2 Use the sDate and eDate parameters to specify clusters in a date range\n",
    "\n",
    "Here we select clusters in the first quarter of 2021.\n",
    "That is in the range: 2021-01-01 to 2021-03-31"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2eb6d08b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37836 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\", searchColName=\"family\", searchValueList=['RaccoonStealer'],\n",
    "                              sDate=\"2021-01-01\", eDate=\"2021-03-31\")\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa11e854",
   "metadata": {},
   "source": [
    "# Step 3 Show all the clusters surrounding a new file using simTlsh\n",
    "\n",
    "We got a file which had TLSH value\n",
    "T14893F844FD459B2FC3D372F6E75C028D763A1FE8A7E630269934BEA023F56D12526911"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cd8680f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 60484 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\",\n",
    "        simTlsh=\"T14893F844FD459B2FC3D372F6E75C028D763A1FE8A7E630269934BEA023F56D12526911\",\n",
    "        simThreshold=130)\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10f2ae84",
   "metadata": {},
   "source": [
    "## Interpretation of Mirai / Gafgyt dendrogram\n",
    "\n",
    "The sample provided is called \"QUERY\" and it is in the 7th row of the dendrogram (near the top).\n",
    "We see that it is close to a Mirai cluster.\n",
    "We can adjust the simThreshold to only display clusters closer to our simTlsh (see below).\n",
    "\n",
    "We see that there is a large branch of Gafgyt malware clusters at the top of the diagram.\n",
    "And a large branch of Mirai malware clusters at the bottom of the diagram.\n",
    "This makes perfect sense.\n",
    "It was reported in April 2021, that Gafgyt had started re-using Mirai code.\n",
    "\n",
    "https://securityaffairs.co/wordpress/116882/cyber-crime/gafgyt-re-uses-mirai-code.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "93e4e615",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19280 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\",\n",
    "        simTlsh=\"T14893F844FD459B2FC3D372F6E75C028D763A1FE8A7E630269934BEA023F56D12526911\",\n",
    "        simThreshold=100)\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2718f0d9",
   "metadata": {},
   "source": [
    "Here we reduced the simThreshold to 80.\n",
    "We find that our simTlsh has a distance < 10 to the Mirai 2021006-30 cluster\n",
    "Our sample is highly likely to be a sample of Mirai malware."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b087ce2",
   "metadata": {},
   "source": [
    "# Step 4 Show all the clusters surrounding a specified cluster\n",
    "\n",
    "We had a SnakeKeyLogger cluster which was first seen on 2021-09-16\n",
    "This cluster has center\n",
    "T12584BF243AFB8019F173AFBA8FE575969B6EFA633603D55D2491038A0613B81CDC153E\n",
    "(this is line 42 of malbaz/cen_389300.csv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "14a38e99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 54916 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\",\n",
    "        simTlsh=\"T12584BF243AFB8019F173AFBA8FE575969B6EFA633603D55D2491038A0613B81CDC153E\",\n",
    "        simThreshold=90)\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58fdb07f",
   "metadata": {},
   "source": [
    "## Interpretation of SnakeKeylogger / AgentTesla / Formbook / Loki dendrogram\n",
    "\n",
    "We see that the SnakeKeyLogger cluster is mixed in a group of AgentTesla / Loki and Formbook clusters.\n",
    "These malware families are known to exhibit similar properties.\n",
    "They are information stealer / RATs which are typically sent attached to spam emails. \n",
    "\n",
    "    \"Researchers say the attackers’ use of several common malware families makes attribution of this \n",
    "    campaign to a particular threat group difficult.\"\n",
    "https://cyberintelmag.com/malware-viruses/year-long-spear-phishing-campaign-targets-energy-sector-with-agent-tesla-other-rats/\n",
    "\n",
    "https://asec.ahnlab.com/en/22074/\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b091815",
   "metadata": {},
   "source": [
    "# Step 5 Work with unlabelled clusters\n",
    "\n",
    "We generate a dendrogram which includes unlabeled clusters.\n",
    "We extract information about those clusters and show how to list the members."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d09850ee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 55676 characters]\n",
      "text/plain": [
       "<Figure size 1080x648 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "\n",
    "(tlist, labels) = tlsh_csvfile(\"malbaz/clust_389300.csv\",\n",
    "        simTlsh=\"T10923013EC661113BCD05DB76E2622B7E24A64C768F6B70D871E7208A3CFE8505F42961\",\n",
    "        simThreshold=180)\n",
    "tlsh_dendrogram(tlist, labelList=labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d44c921d",
   "metadata": {},
   "source": [
    "In the middle of this dendrogram (the green section) we see a group of clusters without labels.\n",
    "We now extract information about Cluster 43300"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1fcd8144",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cluster with cenTlsh=8223F1DD9867B028C6141BB4A2F1759A2D8BAE14EB05409F2CF91B1D330B670F91B7F9\n",
      "showing first  10  samples\n",
      "\tincrease the showN parameter to show more...\n",
      "4df99c597d4fe38e6925652305a4942f367557b8\n",
      "3dd48fd77a14221bd86826b265c09309bcf87b15\n",
      "8e0ea7a7bbcff6ea8afbb97114ef78db33a1c001\n",
      "4268a0d3e90d95381ed07cbbe50a657f8efdf40d\n",
      "b325aa940337112e077106907791b1086ca4cc79\n",
      "c0ddbf7934d778c921b5c4b76851d3623fc2a229\n",
      "728240769d27beee8f1d95a8f68f62af57023e43\n",
      "8e5af4cb33dad1d7b85d131aed48069fa5087955\n",
      "7a6955713284e4a0d62f2c1464ae0683261b300b\n",
      "bc3839eeffa1ec9465d108d82571aabd4f540ace\n"
     ]
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "mb_show_sha1(\"Cluster 43300\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b84953d",
   "metadata": {},
   "source": [
    "We also show how to get more information about Gozi 2020-11-10 (2) which is the 5th row from the bottom."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4b5fc443",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cluster with cenTlsh=T12D13F1567E49D486C008823D9463FDE03876BD39AD7E90977F51B33FA97AA587900037\n",
      "fc2073bfd6d7c8be5b30f0196110056b3ed2b109\n",
      "36a61581ee833366a2f75c900cba601a3b317105\n"
     ]
    }
   ],
   "source": [
    "from pylib.tlsh_lib import *\n",
    "mb_show_sha1(\"MyDoom\", thisDate=\"2021-09-03\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc4c726f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
