Frequently asked questions - SnpEff & SnpSift Documentation
Author: Pablo Cingolani
Description: SnpEff and SnpSift documentation

Skip to content

# SnpSift: Frequently Asked Questions

# Corrupted database VCF files: ClinVar

Some VCF files used as annotation databases can be non-compliant.

Most notably, some ClinVar versions have illegal VCF values, which will make downstream analysis tools, such as `SnpSift` to report the errors.

For example, if you look into the file:

    $ curl -s ftp://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh37/clinvar.vcf.gz | gunzip -c | grep "&base" | head -n 1

    13  32890543    125955  G   A   .   .   ALLELEID=131493;CLNDISDB=MedGen:C2675520,OMIM:612555;CLNDN=Breast-ovarian_cancer,_familial_2;CLNHGVS=NC_000013.10:g.32890543G>A;CLNREVSTAT=no_assertion_criteria_provided;CLNSIG=Uncertain_significance;CLNVC=single_nucleotide_variant;CLNVCSO=SO:0001483;CLNVI=Breast_Cancer_Information_Core__(BRCA2):190-16&base_change=G_to_A;GENEINFO=BRCA2:675;MC=SO:0001627|intron_variant;ORIGIN=1;RS=276174799

As you can see, the "CLNVI" is:

    CLNVI=Breast_Cancer_Information_Core__(BRCA2):190-16&base_change=G_to_A

This means that the CLNVI contains an illegal `'='` character. The VCF specification clearly states that the equal sign is not allowed:

      Reference: https://samtools.github.io/hts-specs/VCFv4.3.pdf
      Section 1.2: "Character encoding, non-printable characters and characters with special meaning"

        Characters with special meaning (such as field delimiters ’;’ in INFO or ’:’ FORMAT fields) must be represented
        using the capitalized percent encoding:
          %3A : (colon)
          %3B ; (semicolon)
          %3D = (equal sign)
          ...

Furthermore, section 1.6.1.8 specifies:

      INFO - additional information: (String, no semi-colons or equals-signs permitted; commas are permitted only
      as delimiters for lists of values; characters with special meaning can be encoded using the percent encoding, see
      Section 1.2; space characters are allowed)

#### Finding all ClinVar problems

An easy way to find many of the problems in the VCF file is to use the `SnpSift checkVcf` command:

    $ java -jar SnpSift.jar vcfCheck clinvar.vcf.gz 2>&1 | head
    ...WARNING: Malformed VCF entryfile '/home/pcingola/Downloads/clinvar.vcf.gz', line 3655:
        Entry  : 1  25717365    17708   C   C   .   .   ALLELEID=32747;CLNDISDB=.;CLNDN=RH_E/e_POLYMORPHISM;CLNHGVS=NC_000001.10:g.25717365C=;CLNREVSTAT=no_assertion_criteria_provided;CLNSIG=Benign;CLNVC=single_nucleotide_variant;CLNVCSO=SO:0001483;CLNVI=OMIM_Allelic_Variant:111700.0001;GENEINFO=RHCE:6006;MC=SO:0001627|intron_variant,SO:0001819|synonymous_variant;ORIGIN=1;RS=609320
        Errors :
            INFO filed 'CLNHGVS' has an invalid value 'NC_000001.10:g.25717365C=' (no spaces, tabs, '=' or ';' are allowed)

    WARNING: Malformed VCF entryfile '/home/pcingola/Downloads/clinvar.vcf.gz', line 3657:
        Entry  : 1  25735202    242743  G   G   .   .   ALLELEID=38411;CLNHGVS=NC_000001.10:g.25735202G=;CLNREVSTAT=no_interpretation_for_the_single_variant;CLNVC=single_nucleotide_variant;CLNVCSO=SO:0001483;CLNVI=OMIM_Allelic_Variant:111700.0002;GENEINFO=RHCE:6006;MC=SO:0001819|synonymous_variant;ORIGIN=1;RS=676785;SSR=1;CLNDISDBINCL=.;CLNDNINCL=RH_C/c_POLYMORPHISM;CLNSIGINCL=17709:Benign
        Errors :
            INFO filed 'CLNHGVS' has an invalid value 'NC_000001.10:g.25735202G=' (no spaces, tabs, '=' or ';' are allowed)

OK, it looks like there are quite a few problems, let's count them:

    $ java -jar SnpSift.jar vcfCheck ~/Downloads/clinvar.vcf.gz 2>&1 | grep WARN | wc -l
    1793

Well, there seems to be 1793 lines VCF with some sort of problem. Let's see how to fix them.

### Fixing ClinVar's VCF database

So, you need to fix ClinVar by either:

1. Remove the offending fields from the VCF file
2. Fix the character coding for the offending values

#### Option 1: Remove the offending fields

This is the easiest way to fix ClinVar's VCF file. First, let's find the corrupted fields:

    $ java -jar SnpSift.jar vcfCheck clinvar.vcf.gz 2>&1 | grep "INFO field" | cut -f 2 -d "'" | sort | uniq -c
        212 CLNHGVS
       1583 CLNVI

OK, there are 212 lines with corrupted `CLNHGVS` fields and 1583 lines with corrupted `CLNVI` fields. Let's create a new database without those fields

    $ java -jar SnpSift.jar rmInfo clinvar.vcf.gz CLNHGVS CLNVI > clinvar.fixed_1.vcf
    00:00:00    Reading STDIN
    00:00:03    Done

    # Let's also compress and index the new file so we can use it as a database
    $ bgzip clinvar.fixed_1.vcf
    $ tabix clinvar.fixed_1.vcf.gz

Now we can re-check the new file to make sure it's OK.

    $ java -jar SnpSift.jar vcfCheck clinvar.fixed_1.vcf.gz
    ....................................................................................................
    100000  ....................................................................................................
    200000  ..................................................................................................

Everything seems OK.

#### Option 2: Fix the encoding

Here we need to fix the encoding of the fields. We know (see previous section) that the problematic fields are `CLNVI` and `CLNHGVS`, so what exactly are the problems?

    $ java -jar SnpSift.jar vcfCheck clinvar.vcf.gz 2>&1 | grep "INFO field" | grep CLNVI | head
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):190-16&base_change=G_to_A' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):190-12&base_change=del_TCT' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):190-5&base_change=del_T' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):190-7&base_change=T_to_C' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):195&base_change=T_to_C' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):203&base_change=G_to_A' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):203&base_change=G_to_C' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):203&base_change=G_to_T' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):214&base_change=A_to_C' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNVI' has an invalid value 'Breast_Cancer_Information_Core__(BRCA2):215&base_change=T_to_C' (no spaces, tabs, '=' or ';' are allowed)

OK, this one seems easy: All we need to do is change `&base_change=` to `&base_change%3D` This can be done with a simple `sed` command:

    $ zcat clinvar.vcf.gz | sed 's/\&base_change=/\&base_change%3D/g'

How about the other field?

    $ java -jar SnpSift.jar vcfCheck clinvar.vcf.gz 2>&1 | grep "INFO field" | grep CLNHGVS | head
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.25717365C=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.25735202G=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.25735306T=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.25735331G=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.94578548T=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.98348885G=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.100672060T=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.114377568A=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.161599571T=' (no spaces, tabs, '=' or ';' are allowed)
            INFO field 'CLNHGVS' has an invalid value 'NC_000001.10:g.161599643T=' (no spaces, tabs, '=' or ';' are allowed)

This is essentially the same, but we need four `sed` commands (one for each base):

    $ zcat clinvar.vcf.gz \
        | sed 's/A=;/A%3D;/' \
        | sed 's/C=;/C%3D;/' \
        | sed 's/G=;/G%3D;/' \
        | sed 's/T=;/T%3D;/'

The fix:

Now, let's put the two previously explained fixes together:

    $ zcat clinvar.vcf.gz \
        | sed 's/\&base_change=/\&base_change%3D/g' \
        | sed 's/A=;/A%3D;/' \
        | sed 's/C=;/C%3D;/' \
        | sed 's/G=;/G%3D;/' \
        | sed 's/T=;/T%3D;/' \
        > clinvar.fixed.vcf

    # Let's also compress and index the new file so we can use it as a database
    $ bgzip clinvar.fixed.vcf
    $ tabix clinvar.fixed.vcf.gz

We re-check the new

    $ java -jar SnpSift.jar vcfCheck clinvar.fixed.vcf.gz
    ....................................................................................................
    100000  ....................................................................................................
    200000  ..................................................................................................

OK, we are done.