Clang Tutorial Part II: LibTooling Example

LibTooling Example

I’ll start with a LibTooling example because I think it’s the most useful interface to Clang, as described in Part I of this tutorial. You can also use this code in the Plugin environment with just a few simple changes. Onwards to the example!

Note: this is a continuation of Clang Tutorial: Part I.

Let’s say that you want to analyze a simple C file, test.c, as shown below:

void do_math(int *x) {
    *x += 5;
}

int main(void) {
    int result = -1, val = 4;
    do_math(&val);
    return result;
}

We’d like to do some simple refactoring/fixes on test.c:

• Change the function do_math to add5, a better name
• Change all calls to do_math to call add5 instead to match the first change
• Fix the return statement to return val instead of result

First, we need to set up some structure. Clang comes packaged with a good Plugin example, but not a LibTooling example. No problem, here’s how to create a LibTooling program.

Here is the full source code for this LibTooling example.

You can check out the repository into the proper directory like so:

$ cd llvm/tools/clang/tools/
$ git clone https://github.com/kevinaboos/LibToolingExample.git example

The following sections are a complete walkthrough of the code in Example.cpp.

Starting with a Main Function

Let’s follow along with the code’s execution flow, starting from the main() function at the bottom of the file:

int main(int argc, const char **argv) {
    // parse the command-line args passed to your code
    CommonOptionsParser op(argc, argv);
    // create a new Clang Tool instance (a LibTooling environment)
    ClangTool Tool(op.getCompilations(), op.getSourcePathList());

    // run the Clang Tool, creating a new FrontendAction (explained below)
    int result = Tool.run(newFrontendActionFactory<ExampleFrontendAction>());

    errs() << "\nFound " << numFunctions << " functions.\n\n";
    // print out the rewritten source code ("rewriter" is a global var.)
    rewriter.getEditBuffer(rewriter.getSourceMgr().getMainFileID()).write(errs());
    return result;
}

Everything in main() is fairly self-explanatory, except rewriter, which is explained later. You’re just setting up a Clang Tool (L5), passing it command-line arguments (op.getCompilations()) and the list of source files (op.getSourcePathList()), and then running it (L8). The benefit of using LibTooling is that you can do things before (L6) and after (L9) the analysis, like printing out the modified code in L12 or counting the number of functions in several source files (L10). You cannot do this with a Clang Plugin.We also need a few global variables (at the top of the file):

Rewriter rewriter;
int numFunctions = 0;

Creating a FrontendAction

Now, let’s create our own custom FrontendAction, which is merely a class that will perform some action in the Clang frontend. We’ll choose an ASTFrontendAction because we want to analyze the AST representation of the source code test.c.

class ExampleFrontendAction : public ASTFrontendAction {
public:
    virtual ASTConsumer *CreateASTConsumer(CompilerInstance &CI, StringRef file) {
        return new ExampleASTConsumer(&CI); // pass CI pointer to ASTConsumer
    }
};

Nothing too complex going on here. We just create a subclass of ASTFrontendAction and overwrite the CreateASTConsumer function so that we can return our own ASTConsumer, as shown below. We also pass a pointer to the CompilerInstance because it contains a lot of contextual information that we’ll need later during our actual analysis.

Creating an ASTConsumer

The ASTConsumer “consumes” (reads) the AST produced by the Clang parser. You can override however many functions you wish, so that your code will be called when a certain type of AST item has been parsed. First, let’s override HandleTopLevelDecl(), which will be called whenever Clang parses a new set of top-level declarations (such as global variables, function definitions, etc.)

class ExampleASTConsumer : public ASTConsumer {
private:
    ExampleVisitor *visitor; // doesn't have to be private

public:
    // override the constructor in order to pass CI
    explicit ExampleASTConsumer(CompilerInstance *CI)
        : visitor(new ExampleVisitor(CI)) // initialize the visitor
        { }

    // override this to call our ExampleVisitor on each top-level Decl
    virtual bool HandleTopLevelDecl(DeclGroupRef DG) {
        // a DeclGroupRef may have multiple Decls, so we iterate through each one
        for (DeclGroupRef::iterator i = DG.begin(), e = DG.end(); i != e; i++) {
            Decl *D = *i;
            visitor->TraverseDecl(D); // recursively visit each AST node in Decl "D"
        }
        return true;
    }
};

Basically, this code uses our ExampleVisitor (described below) to visit the AST nodes in each top-level declaration in the entire source file. For test.c, two Decls would be visited, the FunctionDecl for do_math() and the FunctionDecl for main().

A Better Implementation of ASTConsumer

However, overriding HandleTopLevelDecl() means that your code in that function will be immediately called each time a new Decl is parsed in the source, not after the entire source file has been parsed. This creates a problem because, from the parser’s point of view, when do_math() is being visited, it is completely unaware that main() exists. This means you can’t access or reason about functions defined after the function you’re currently analyzing.  
★ ★ hey, that’s important! ★ ★

Luckily, the ASTConsumer class has a better function to override, HandleTranslationUnit(), which is called only after the entire source file is parsed. In this case, a translation unit effectively represents an entire source file. An ASTContext class is used to represent the AST for that source file, and it has a ton of useful members (go read about it!).

So, instead of overriding HandleTopLevelDecl(), let’s go with HandleTranslationUnit() because it works more like we would expect.

    // this replaces "HandleTopLevelDecl"
    // override this to call our ExampleVisitor on the entire source file
    virtual void HandleTranslationUnit(ASTContext &Context) {
        /* we can use ASTContext to get the TranslationUnitDecl, which is
           a single Decl that collectively represents the entire source file */
        visitor->TraverseDecl(Context.getTranslationUnitDecl());
    }

For the most part, you should use HandleTranslationUnit(), especially when you pair it with a RecursiveASTVisitor, like we do below.

Creating a RecursiveASTVisitor

At long last we’re ready to get some real work done. The previous two sections were just to set up infrastructure.The RecursiveASTVisitor is a fascinating class with more to it than meets the eye. It allows you to Visit any type of AST node, such as FunctionDecl and Stmt, simply by overriding a function with that name, e.g., VisitFunctionDecl and VisitStmt. This same format works with any AST class. Clang also offers an official tutorial on this, and while complete, it’s very brief.

For such Visit* functions, you must return true to continue traversing the AST (examining other nodes) and return false to halt the traversal entirely and essentially exit Clang. You shouldn’t ever call any of the Visit* functions directly; instead call TraverseDecl (like we did in our ExampleASTConsumer above), which will call the correct Visit* function behind the scenes.

Based on our objective of rewriting function definitions and statements, we only need to override VisitFunctionDecl and VisitStmt. Here’s how to do so:

class ExampleVisitor : public RecursiveASTVisitor {
private:
    ASTContext *astContext; // used for getting additional AST info

public:
    explicit ExampleVisitor(CompilerInstance *CI)
        : astContext(&(CI->getASTContext())) // initialize private members
    {
        rewriter.setSourceMgr(astContext->getSourceManager(),
            astContext->getLangOpts());
    }

    virtual bool VisitFunctionDecl(FunctionDecl *func) {
        numFunctions++;
        string funcName = func->getNameInfo().getName().getAsString();
        if (funcName == "do_math") {
            rewriter.ReplaceText(func->getLocation(), funcName.length(), "add5");
            errs() << "** Rewrote function def: " << funcName << "\n";         
        }         
        return true;     
    }     
    
    virtual bool VisitStmt(Stmt *st) {
        if (ReturnStmt *ret = dyn_cast(st)) {
            rewriter.ReplaceText(ret->getRetValue()->getLocStart(), 6, "val");
            errs() << "** Rewrote ReturnStmt\n";
        }
        if (CallExpr *call = dyn_cast(st)) {
            rewriter.ReplaceText(call->getLocStart(), 7, "add5");
            errs() << "** Rewrote function call\n";
        }
        return true;
    }
};

The above code introduces the Rewriter class, which lets you make textual changes to the source code. It is commonly used for refactoring or making small code changes. We also used it at the end of our program’s main() function to print out the full modified source code.

Using Rewriter means that you need to find the correct SourceLocation to insert/replace text. Understanding which location to choose (getLocation(), getLocStart(), etc) can be difficult, so I’ve explained several common types of location getters in this post.

Also, note the use of dyn_cast (L24, L28) to check whether the Stmt st is a ReturnStmt or a CallExpr. Click to read more about dyn_cast.

Finally, errs() is simply the stderr stream used throughout LLVM/Clang to print debugging info.

Getting More Specific with Visit* Functions

In this example, we only need to modify return statements (L8 of test.c) and function calls (L7 of test.c). Therefore, instead of overriding VisitStmt (more generic), we can override VisitReturnStmt and VisitCallExpr (more specific). Both ReturnStmt and CallExpr are subclasses of Stmt. This is the beauty of Clang’s AST and the RecursiveASTVisitor class — it can be as generic or as specific as you need. Here’s what that more specific code would look like:

    // this replaces the VisitStmt function above
    virtual bool VisitReturnStmt(ReturnStmt *ret) {
        rewriter.ReplaceText(ret->getLocStart(), 6, "val");
        errs() << "** Rewrote ReturnStmt\n";
        return true;
    }
    virtual bool VisitCallExpr(CallExpr *call) {
        rewriter.ReplaceText(call->getLocStart(), 7, "add5");
        errs() << "** Rewrote function call\n";
        return true;
    }

Putting It All Together and Compiling the Analysis

Again, I have posted the full code for this example, including all #include directives and global variables, as well as a Makefile and instructions for compiling and running it. Clang makefiles can be complex, but I won’t bother describing them because there’s some fairly good documentation on Clang Makefiles. Here’s the full Makefile necessary for building this example:

CLANG_LEVEL := ../..

TOOLNAME = example  #the name of your tool's executable

SOURCES := Example.cpp  #the Clang source files you want to compile

include $(CLANG_LEVEL)/../../Makefile.config

LINK_COMPONENTS := $(TARGETS_TO_BUILD) asmparser bitreader support mc option

USEDLIBS = clangFrontend.a clangSerialization.a clangDriver.a \
           clangTooling.a clangParse.a clangSema.a \
           clangAnalysis.a clangRewriteFrontend.a clangRewriteCore.a \
           clangEdit.a clangAST.a clangLex.a clangBasic.a

include $(CLANG_LEVEL)/Makefile

The above Makefile only works if you keep your sources in the llvm/tools/clang/tools directory. For example, I created a directory at  llvm/tools/clang/tools/example/ for placing the example C++ file and Makefile.

Running the Analysis

At the top of this page, I showed a simple C file, test.c, that we’ll use as the input to our analysis. It’s easiest to run our analysis with a quick shell script:

#!/bin/bash

LLVM_DIR=~/static_analysis/llvm/  #the location of your llvm dir
$LLVM_DIR/Debug+Asserts/bin/example test.c --

You can run this script from anywhere, as long as it’s in the same directory as your test.c file.

Perhaps you’d like to run your LibTooling analysis on multiple source files, or use some additional CFLAGS (or GCC options), or even pass your own command-line arguments to your Clang code. The script below has examples of all three:

#!/bin/bash

LLVM_DIR=~/static_analysis/llvm/  #the location of your llvm dir
$LLVM_DIR/Debug+Asserts/bin/example \
    file1.c file2.c file3.c \
    -myCmdLineArg argument1 \
    -- \
    -Wall -Isome/include/dir

Note that anything before the double dash “--” on L6 is an input to your LibTooling program, argv in main(), while anything after the double dash is an input to Clang itself (you won’t concern yourself with those).

If you want to learn how to parse command-line arguments such as “-myCmdLineArg argument1” above, then check out the LibTooling section of my post Clang: Handling Command-Line Arguments (coming soon!).

Conclusion

Assuming you were able to run your analysis script, congratulations on your first foray into Clang! You should see the following output:

** Rewrote function def: do_math
** Rewrote function call
** Rewrote ReturnStmt

Found 2 functions.

void add5(int *x) {
    *x += 5;
}

int main(void) {
    int result = -1, val = 4;
    add5(&val);
    return val;
}

I hope you enjoyed working through this Clang LibTooling example. Feel free to post questions or follow-ups in the comment section; I’m always willing to add things to the tutorial.

Up next: how to use the Clang Plugin interface.

→ Clang Tutorial Part III: Plugin Example.