Building Plugins

Rebar3’s system is based on the concept of providers. A provider has three callbacks:

• init(State) -> {ok, NewState}, which helps set up the state required, state dependencies, etc.
• do(State) -> {ok, NewState} | {error, Error}, which does the actual work.
• format_error(Error) -> String, which prints errors when they happen, and to filter out sensitive elements from the state.

A provider should also be an OTP Library application, which can be fetched as any other Erlang dependency, except for Rebar3 rather than your own system or application.

This document contains the following elements:

• Using a Plugin
• Reference
  • Provider Interface
  • List of Possible Dependencies
  • Rebar API
  • Rebar State Manipulation
  • Namespaces
• Tutorial
  • First Version
  • Optionally Search Deps

Using a Plugin

To use the a plugin, add it to the rebar.config:

{plugins, [
  {plugin_name, {git, "git@host:user/name-of-plugin.git", {tag, "1.0.0"}}}
]}.

Then you can just call it directly:

$ rebar3 plugin_name
===> Fetching plugin_name
===> Compiling plugin_name
<PLUGIN OUTPUT>

Reference

Provider Interface

Each provider has the following options available:

• name: The ‘user friendly’ name of the task.
• module: The module implementation of the task.
• hooks: A two-tuple of provider names for pre and post-hooks ({Pre, Post}).
• bare: Indicates whether task can be run by users or not. Should be true.
• deps: The list of dependencies, providers that need to run before this one. You do not need to include the dependencies of your dependencies.
• desc: The description for the task, used by rebar3 help
• short_desc: A one line short description of the task, used in lists of providers
• example: An example of the task usage, such as "rebar3 my-provider args"
• opts: The list of options that the task requires/understands. The form of each option is {Key, $Character, "StringName", Spec, HelpText}, where:
  • Key is an atom, to be used to fetch the value later;
  • $Character is the short form of the option. So if the command is to be entered as a -c Arg, $c is the value of this field
  • Spec is either a type (atom, binary, boolean, float, integer, or string), a type with a default value ({Type, Val}), or the atom undefined.
• profiles: Profiles to use for provider. Default to [default].
• namespace: namespace the provider is registered in. Defaults to default, which is the main namespace.

These options are to be added to the provider when creating it.

A provider has the following implementation:

-module(provider_template).
-behaviour(provider).

-export([init/1, do/1, format_error/1]).

%% ===================================================================
%% Public API
%% ===================================================================

%% Called when rebar3 first boots, before even parsing the arguments
%% or commands to be run. Purely initiates the provider, and nothing
%% else should be done here.
-spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
init(State) ->
    Provider = providers:create([Options]),
    {ok, rebar_state:add_provider(State, Provider)}.

%% Run the code for the plugin. The command line argument are parsed
%% and dependencies have been run.
-spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.
do(State) ->
    {ok, State}.

%% When an exception is raised or a value returned as
%% `{error, {?MODULE, Reason}}` will see the `format_error(Reason)`
%% function called for them, so a string can be formatted explaining
%% the issue.
-spec format_error(any()) -> iolist().
format_error(Reason) ->
    io_lib:format("~p", [Reason]).

List of Possible Dependencies

All dependencies are in the default namespace until indicated otherwise

Note that you can depend on more than one provider, but they must be in the same namespace

Rebar API

Rebar comes with a module called rebar_api exporting commonly needed functions when writing providers. Functions include:

Do note that all logging functions automatically add a new line (~n) to every expression logged.

Rebar State Manipulation

The State argument passed to the plugin provider can be operated on with the rebar_state module through the following interface:

Manipulate Application State

Each application being built (project applications and dependencies). All AppInfo records can be found in the State and accessed through project_apps/1 and all_deps/1

Namespaces

For plugins that might require multiple commands all adapted to a single type of task (such as implementing a suite of tools for a BEAM language other than Erlang), rather than having multiple commands polluting the command space or requiring prefixes such as rebar3 mylang_compile, rebar3 introduces support for namespaces.

A plugin can be declared to belong to a given namespace. For example, the ErlyDTL compiler plugin introduces the compile command under the erlydtl namespace. It can therefore be invoked as rebar3 erlydtl compile. If the erlydtl namespace had other commands such as clean, they could be chained as rebar3 erlydtl clean, compile.

In other ways, a namespace acts like do (rebar3 do compile, edoc), but operating on a non-default set of commands.

To declare a namespace, an provider needs only to use the {namespace, Namespace} option in its configuration list. The provider will automatically register the new namespace and be available under this term.

🚧

Namespaces also apply to provider dependencies and hooks

If a provider is part of a given namespace, its dependencies will be searched within that same namespace. Therefore if rebar3 mytool rebuild depends on compile, the compile command will be looked for in the mytool namespace.

To use the default compile command, the dependency must be declared as {default, compile}, or more generally {NameSpace, Command}.

The same mechanism is applied for hooks.

Tutorial

First version

In this tutorial, we’ll show how to start from scratch, and get a basic plugin written. The plugin will be quite simple: it will look for instances of TODO: lines in comments and report them as warnings. The final code for the plugin can be found on bitbucket.

The first step is to create a new OTP Application that will contain the plugin:

→ rebar3 new plugin todo desc="example rebar3 plugin"
...
→ cd todo
→ git init
Initialized empty Git repository in /Users/ferd/code/self/todo/.git/

The src/todo.erl file will be used to call the initialization of all commands. For now we’ll only have one todo command. Open up the src/todo_prv.erl file that will contain the command implementation, and make sure you have the following skeleton in place:

-module(todo_prv).
-behaviour(provider).

-export([init/1, do/1, format_error/1]).

-define(PROVIDER, todo).
-define(DEPS, [app_discovery]).

%% ===================================================================
%% Public API
%% ===================================================================
-spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
init(State) ->
    Provider = providers:create([
            {name, ?PROVIDER},          % The 'user friendly' name of the task
            {module, ?MODULE},          % The module implementation of the task
            {bare, true},               % The task can be run by the user, always true
            {deps, ?DEPS},              % The list of dependencies
            {example, "rebar provider_todo"}, % How to use the plugin
            {opts, []}                  % list of options understood by the plugin
            {short_desc, "example rebar3 plugin"},
            {desc, ""}
    ]),
    {ok, rebar_state:add_provider(State, Provider)}.


-spec do(rebar_state:t()) -> {ok, rebar_state:t()} | {error, string()}.
do(State) ->
    {ok, State}.

-spec format_error(any()) -> iolist().
format_error(Reason) ->
    io_lib:format("~p", [Reason]).

This shows all the basic content needed. Note that we leave the DEPS macro to the value app_discovery, used to mean that the plugin should at least find the project’s source code (excluding dependencies).

In this case, we need to change very little in init/1. Here’s the new provider description:

Provider = providers:create([
            {name, ?PROVIDER},       % The 'user friendly' name of the task
            {module, ?MODULE},       % The module implementation of the task
            {bare, true},            % The task can be run by the user, always true
            {deps, ?DEPS},           % The list of dependencies
            {example, "rebar todo"}, % How to use the plugin
            {opts, []},              % list of options understood by the plugin
            {short_desc, "Reports TODOs in source code"},
            {desc, "Scans top-level application source and find "
                   "instances of TODO: in commented out content "
                   "to report it to the user."}
    ]),

Instead, most of the work will need to be done directly in do/1. We’ll use the rebar_state module to fetch all the applications we need. This can be done by calling the project_apps/1 function, which returns the list of the project’s top-level applications.

do(State) ->
    lists:foreach(fun check_todo_app/1, rebar_state:project_apps(State)),
    {ok, State}.

This, on a high level, means that we’ll check each top-level app one at a time (there may often be more than one top-level application when working with releases)

The rest is filler code specific to the plugin, in charge of reading each app path, go read code in there, and find instances of ‘TODO:’ in comments in the code:

check_todo_app(App) ->
    Paths = rebar_dir:src_dirs(rebar_app_info:opts(App)),
    Mods = find_source_files(Paths),
    case lists:foldl(fun check_todo_mod/2, [], Mods) of
        [] -> ok;
        Instances -> display_todos(rebar_app_info:name(App), Instances)
    end.
find_source_files(Paths) ->
    find_source_files(Paths, []).

find_source_files([], Files) ->
    Files;
find_source_files([Path | Rest], Files) ->
    find_source_files(Rest, [filename:join(Path, Mod) || Mod <- filelib:wildcard("*.erl", Path)] ++ Files).

check_todo_mod(ModPath, Matches) ->
    {ok, Bin} = file:read_file(ModPath),
    case find_todo_lines(Bin) of
        [] -> Matches;
        Lines -> [{ModPath, Lines} | Matches]
    end.

find_todo_lines(File) ->
    case re:run(File, "%+.*(TODO:.*)", [{capture, all_but_first, binary}, global, caseless]) of
        {match, DeepBins} -> lists:flatten(DeepBins);
        nomatch -> []
    end.

display_todos(_, []) -> ok;
display_todos(App, FileMatches) ->
    io:format("Application ~s~n",[App]),
    [begin
      io:format("\t~s~n",[Mod]),
      [io:format("\t  ~s~n",[TODO]) || TODO <- TODOs]
     end || {Mod, TODOs} <- FileMatches],
    ok.

Just using io:format/2 to output is going to be fine.

To test the plugin, push it to a source repository somewhere. Pick one of your projects, and add something to the rebar.config:

{plugins, [
  {todo, {git, "git@bitbucket.org:ferd/rebar3-todo-plugin.git", {branch, "master"}}}
]}.

Then you can just call it directly:

→ rebar3 todo

===> Fetching todo

===> Compiling todo

Application merklet

    /Users/ferd/code/self/merklet/src/merklet.erl

      todo: consider endianness for absolute portability

Rebar3 will download and install the plugin, and figure out when to run it. Once compiled, it can be run at any time again.

Optionally Search Deps

Let’s extend things a bit. Maybe from time to time (when cutting a release), we’d like to make sure none of our dependencies contain ‘TODO:’s either.

To do this, we’ll need to go parse command line arguments a bit, and change our execution model. The ?DEPS macro will now need to specify that the todo provider can only run after dependencies have been installed:

-define(DEPS, [install_deps]).

We can add the option to the list we use to configure the provider in init/1:

{opts, [                 % list of options understood by the plugin
    {deps, $d, "deps", undefined, "also run against dependencies"}
]},

And then we can implement the switch to figure out what to search:

do(State) ->
    Apps = case discovery_type(State) of
        project -> rebar_state:project_apps(State);
        deps -> rebar_state:project_apps(State) ++ lists:usort(rebar_state:all_deps(State))
    end,
    lists:foreach(fun check_todo_app/1, Apps),
    {ok, State}.

[...]

discovery_type(State) ->
    {Args, _} = rebar_state:command_parsed_args(State),
    case proplists:get_value(deps, Args) of
        undefined -> project;
        _ -> deps
    end.

The deps option is found using rebar_state:command_parsed_args(State), which will return a proplist of terms on the command-line after ‘todo’, and will take care of validating whether the flags are accepted or not. The rest can remain the same.

Push the new code for the plugin, and try it again on a project with dependencies:

===> Fetching todo

===> Compiling todo

===> Fetching bootstrap

===> Fetching file_monitor

===> Fetching recon

[...]

Application dirmon

    /Users/ferd/code/self/figsync/apps/dirmon/src/dirmon_tracker.erl

      TODO: Peeranha should expose the UUID from a node.

Application meck

    /Users/ferd/code/self/figsync/_deps/meck/src/meck_proc.erl

      TODO: What to do here?

      TODO: What to do here?

Rebar3 will now go pick dependencies before running the plugin on there.

you can also see that the help will be completed for you:

→ rebar3 help todo

Scans top-level application source and find instances of TODO: in commented out content to report it to the user.

Usage: rebar todo [-d]

That’s it, the todo plugin is now complete! It’s ready to ship and be included in other repositories.

Adding More Commands

To add more commands to the same plugin, simply add entries to the init function in the main module:

-module(todo).

-export([init/1]).

-spec init(rebar_state:t()) -> {ok, rebar_state:t()}.
init(State) ->
    %% initialize all commands here
    {ok, State1} = todo_prv:init(State),
    {ok, State2} = todo_other_prv:init(State1),
    {ok, State2}.

And Rebar3 will pick it up from there.