README: Add newline after code-block.

This is the second 'release' for Puppetboard introducing some new
features and compatibility with PuppetDB 1.5 / v3 API.

CHANGELOG.rst

@@ -0,0 +1,41 @@
+#########
+Changelog
+#########
+
+0.0.2
+=====
+In this release we've introduced a few new things. First of all we now require
+pypuppetdb version 0.0.4 or later which includes support for the v3 API
+introduced with PuppetDB 1.5.
+
+Because of changes in PuppetDB 1.5 and therefor in pypuppetdb users of the v2
+API, regardless of the PuppetDB version, will no longer be able to view reports
+or events.
+
+In light of this the following settings have been removed:
+
+* ``PUPPETDB_EXPERIMENTAL``
+
+Two new settings have been added:
+
+* ``PUPPETDB_API``: an integer, defaulting to ``3``, representing the API
+  version we want to use.
+* ``ENABLE_QUERY``: a boolean, defaulting to ``True``, on wether or not to
+  be able to use the Query tab.
+
+We've also added a few new features:
+
+* Thanks to some work done during PuppetConf together with Nick Lewis (from
+  Puppet Labs) we now expose all of PuppetDB's metrics in the Metrics tab. The
+  formatting isn't exactly pretty but it's a start.
+* Spencer Krum added the graphing capabilities to the Facts tab.
+* Daniel Lawrence added a feature so that facts on the node view are clickable
+  and take you to the complete overview of that fact for your infrastructure
+  and made the nodes in the complete facts list clickable so you can jump to a
+  node.
+* Klavs Klavsen contributed some documentation on how to run Puppetboard with
+  Passenger.
+
+0.0.1
+=====
+Initial release.

README.rst

@@ -17,12 +17,14 @@ Because this project is powered by Flask we are restricted to:
     * Python 2.6
     * Python 2.7
 
-.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/node-experimental.png
+.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/node-v3.png
    :alt: View of a node
    :width: 1024
    :height: 700
    :align: center
 
+.. contents::
+
 Word of caution
 ===============
 
@@ -66,6 +68,7 @@ and ``uwsgi`` can deal with.
 
   * Apache mod_wsgi configuration: http://flask.pocoo.org/docs/deploying/mod_wsgi/
   * uwsgi configuration: ``uwsgi --http :9090 --wsgi-file /path/to/puppetboard/wsgi.py``
+  * Passenger
 
 In the case of uwsgi you'll of course need something like nginx in front of it to
 proxy the requests to it.
@@ -74,6 +77,33 @@ Don't forget that you also need to serve the ``static/`` folder on the
 ``/static`` URL of your vhost. (I'm considering embedding the little additional
 Javascript and CSS this application has so no one has to bother with that).
 
+Passenger
+^^^^^^^^^
+From within the Puppetboard checkout:
+
+.. code-block:: bash
+
+   mkdir public
+   mkdir tmp
+   ln -s wsgi.py passenger_wsgi.py
+
+The apache vhost configuration:
+
+.. code-block::
+
+   <VirtualHost *:80>
+       ServerName puppetboard.example.tld
+       DocumentRoot /path/to/puppetboard/public
+
+       RackAutoDetect On
+       Alias /static /path/to/puppetboard/static
+       <Directory /path/to/puppetboard/>
+                Options None
+                Order allow,deny
+                allow from all
+       </Directory>
+   </VirtualHost>
+
 Configuration
 =============
 
@@ -93,6 +123,42 @@ PuppetDB's experimental API endpoints.
 However, if you haven't enabled them for Puppet it isn't particularily
 useful to enable them here as there will be no data to retrieve.
 
+Getting Help
+============
+This project is still very new so it's not inconceivable you'll run into
+issues.
+
+For bug reports you can file an `issue`_. If you need help with something
+feel free to hit up `@daenney`_ by e-mail or find him on IRC. He can usually
+be found on `IRCnet`_ and `Freenode`_ and idles in #puppet.
+
+There's now also the #puppetboard channel on `Freenode`_ where we hang out
+and answer questions related to pypuppetdb and Puppetboard.
+
+.. _issue: https://github.com/nedap/puppetboard/issues
+.. _@daenney: https://github.com/daenney
+.. _IRCnet: http://www.ircnet.org
+.. _Freenode: http://freenode.net
+
+Third party
+===========
+Some people have already started building things with and around Puppetboard.
+
+`Hunter Haugen`_ has provided a Vagrant setup:
+
+* https://github.com/hunner/puppetboard-vagrant
+
+`Spencer Krum`_ created a Puppet module to install Puppetboard with:
+
+* https://github.com/nibalizer/puppet-module-puppetboard
+
+You can install it with:
+
+    puppet module install nibalizer-puppetboard
+
+.. _Hunter Haugen: https://github.com/hunner
+.. _Spencer Krum: https://github.com/nibalizer
+
 Contributing
 ============
 We welcome contributions to this project. However, there are a few ground
@@ -152,22 +218,34 @@ Screenshots
    :height: 700
    :align: center
 
+.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/metrics.png
+   :alt: Query view
+   :width: 1024
+   :height: 700
+   :align: center
+
+.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/metric.png
+   :alt: Query view
+   :width: 1024
+   :height: 700
+   :align: center
+
 .. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/query.png
    :alt: Query view
    :width: 1024
    :height: 700
    :align: center
 
-With experimental endpoints
+API v3
----------------------------
+------
-
+           
-.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/nodes-experimental.png
+.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/nodes-v3.png
    :alt: Nodes table with experimental endpoints enabled
    :width: 1024
    :height: 700
    :align: center
 
-.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/node-experimental.png
+.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/node-v3.png
    :alt: Node view with experimental endpoints enabled
    :width: 1024
    :height: 700
@@ -178,18 +256,3 @@ With experimental endpoints
    :width: 1024
    :height: 700
    :align: center
-
-Error page
-----------
-
-.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/no-experimental.png
-   :alt: Accessing disabled experimental feature
-   :width: 1024
-   :height: 700
-   :align: center
-
-.. image:: https://raw.github.com/nedap/puppetboard/master/screenshots/broken.png
-   :alt: Error message
-   :width: 1024
-   :height: 700
-   :align: center

dev.py

@@ -2,7 +2,8 @@ from __future__ import unicode_literals
 from __future__ import absolute_import
 
 from puppetboard.app import app
+from puppetboard.default_settings import DEV_LISTEN_HOST, DEV_LISTEN_PORT
 
 if __name__ == '__main__':
     app.debug=True
-    app.run('127.0.0.1')
+    app.run(DEV_LISTEN_HOST, DEV_LISTEN_PORT)

puppetboard/app.py

@@ -4,6 +4,7 @@ from __future__ import absolute_import
 import os
 import logging
 import collections
+import urllib
 
 from flask import (
     Flask, render_template, abort, url_for,
@@ -11,7 +12,6 @@ from flask import (
     )
 
 from pypuppetdb import connect
-from pypuppetdb.errors import ExperimentalDisabledError
 
 from puppetboard.forms import QueryForm
 from puppetboard.utils import (
@@ -26,13 +26,13 @@ app.config.from_envvar('PUPPETBOARD_SETTINGS', silent=True)
 app.secret_key = os.urandom(24)
 
 puppetdb = connect(
+        api_version=app.config['PUPPETDB_API'],
         host=app.config['PUPPETDB_HOST'],
         port=app.config['PUPPETDB_PORT'],
         ssl=app.config['PUPPETDB_SSL'],
         ssl_key=app.config['PUPPETDB_KEY'],
         ssl_cert=app.config['PUPPETDB_CERT'],
-        timeout=app.config['PUPPETDB_TIMEOUT'],
+        timeout=app.config['PUPPETDB_TIMEOUT'],)
-        experimental=app.config['PUPPETDB_EXPERIMENTAL'])
 
 numeric_level = getattr(logging, app.config['LOGLEVEL'].upper(), None)
 if not isinstance(numeric_level, int):
@@ -51,13 +51,18 @@ def stream_template(template_name, **context):
 def bad_request(e):
     return render_template('400.html'), 400
 
+@app.errorhandler(403)
+def bad_request(e):
+    return render_template('403.html'), 400
+
 @app.errorhandler(404)
 def not_found(e):
     return render_template('404.html'), 404
 
 @app.errorhandler(412)
 def precond_failed(e):
-    """We're slightly abusing 412 to handle ExperimentalDisabled errors."""
+    """We're slightly abusing 412 to handle missing features
+    depending on the API version."""
     return render_template('412.html'), 412
 
 @app.errorhandler(500)
@@ -83,9 +88,9 @@ def index():
     metrics = {
       'num_nodes': num_nodes['Value'],
       'num_resources': num_resources['Value'],
-      'avg_resources_node': "{:10.6f}".format(avg_resources_node['Value']),
+      'avg_resources_node': "{0:10.6f}".format(avg_resources_node['Value']),
       'mean_failed_commands': mean_failed_commands['MeanRate'],
-      'mean_command_time': "{:10.6f}".format(mean_command_time['MeanRate']),
+      'mean_command_time': "{0:10.6f}".format(mean_command_time['MeanRate']),
       }
     return render_template('index.html', metrics=metrics)
 
@@ -111,7 +116,7 @@ def node(node_name):
     """
     node = get_or_abort(puppetdb.node, node_name)
     facts =  node.facts()
-    if app.config['PUPPETDB_EXPERIMENTAL']:
+    if app.config['PUPPETDB_API'] > 2:
         reports = ten_reports(node.reports())
     else:
         reports = iter([])
@@ -122,21 +127,21 @@ def node(node_name):
 def reports():
     """Doesn't do much yet but is meant to show something like the reports of
     the last half our, something like that."""
-    if app.config['PUPPETDB_EXPERIMENTAL']:
+    if app.config['PUPPETDB_API'] > 2:
         return render_template('reports.html')
     else:
-        log.warn('Access to experimental endpoint not allowed.')
+        log.warn('PuppetDB API prior to v3 cannot access reports.')
         abort(412)
 
 @app.route('/reports/<node>')
 def reports_node(node):
     """Fetches all reports for a node and processes them eventually rendering
     a table displaying those reports."""
-    if app.config['PUPPETDB_EXPERIMENTAL']:
+    if app.config['PUPPETDB_API'] > 2:
         reports = ten_reports(yield_or_stop(
             puppetdb.reports('["=", "certname", "{0}"]'.format(node))))
     else:
-        log.warn('Access to experimental endpoint not allowed.')
+        log.warn('PuppetDB API prior to v3 cannot access reports.')
         abort(412)
     return render_template('reports_node.html', reports=reports,
             nodename=node)
@@ -145,10 +150,10 @@ def reports_node(node):
 def report(node, report_id):
     """Displays a single report including all the events associated with that
     report and their status."""
-    if app.config['PUPPETDB_EXPERIMENTAL']:
+    if app.config['PUPPETDB_API'] > 2:
         reports = puppetdb.reports('["=", "certname", "{0}"]'.format(node))
     else:
-        log.warn('Access to experimental endpoint not allowed.')
+        log.warn('PuppetDB API prior to v3 cannot access reports.')
         abort(412)
 
     for report in reports:
@@ -179,9 +184,12 @@ def facts():
 def fact(fact):
     """Fetches the specific fact from PuppetDB and displays its value per
     node for which this fact is known."""
+    # we can only consume the generator once, lists can be doubly consumed
+    # om nom nom
+    localfacts = [ f for f in yield_or_stop(puppetdb.facts(name=fact)) ]
     return Response(stream_with_context(stream_template('fact.html',
         name=fact,
-        facts=yield_or_stop(puppetdb.facts(name=fact)))))
+        facts=localfacts)))
 
 @app.route('/query', methods=('GET', 'POST'))
 def query():
@@ -190,9 +198,26 @@ def query():
     of the possible exceptions are being handled just yet. This will return
     the JSON of the response or a message telling you what whent wrong /
     why nothing was returned."""
-    form = QueryForm()
+    if app.config['ENABLE_QUERY']:
-    if form.validate_on_submit():
+      form = QueryForm()
-        result = get_or_abort(puppetdb._query, form.endpoints.data,
+      if form.validate_on_submit():
-                query='[{0}]'.format(form.query.data))
+          result = get_or_abort(puppetdb._query, form.endpoints.data,
-        return render_template('query.html', form=form, result=result)
+                  query='[{0}]'.format(form.query.data))
-    return render_template('query.html', form=form)
+          return render_template('query.html', form=form, result=result)
+      return render_template('query.html', form=form)
+    else:
+        log.warn('Access to query interface disabled by administrator..')
+        abort(403)
+
+@app.route('/metrics')
+def metrics():
+  metrics = puppetdb._query('metrics', path='mbeans')
+  for key,value in metrics.iteritems():
+    metrics[key]=value.split('/')[3]
+  return render_template('metrics.html', metrics=sorted(metrics.items()))
+
+@app.route('/metric/<metric>')
+def metric(metric):
+  name = urllib.unquote(metric)
+  metric = puppetdb.metric(metric)
+  return render_template('metric.html', name=name, metric=sorted(metric.items()))

puppetboard/default_settings.py

@@ -4,5 +4,8 @@ PUPPETDB_SSL=False
 PUPPETDB_KEY=None
 PUPPETDB_CERT=None
 PUPPETDB_TIMEOUT=20
-PUPPETDB_EXPERIMENTAL=False
+PUPPETDB_API=3
+DEV_LISTEN_HOST='127.0.0.1'
+DEV_LISTEN_PORT=5000
+ENABLE_QUERY=True
 LOGLEVEL='info'

puppetboard/templates/403.html

@@ -0,0 +1,11 @@
+{% extends 'layout.html' %}
+{% block row_fluid %}
+<div class="container" style="margin-bottom:55px;">
+  <div class="row">
+    <div class="span12">
+      <h2>Permission Denied</h2>
+      <p>What you were looking for has been disabled by the administrator.</p>
+    </div>
+  </div>
+</div>
+{% endblock %}

puppetboard/templates/412.html

@@ -3,8 +3,8 @@
 <div class="container" style="margin-bottom:55px;">
   <div class="row">
     <div class="span12">
-      <h2>Experimental Disabled</h2>
+      <h2>Feature unavailable</h2>
-      <p>You're trying to access a feature restricted to PuppetDB's Experimental API but haven't configured Puppetboard to allow this.</p>
+      <p>You've configured Puppetboard with an API version that does not support this feature.</p>
     </div>
   </div>
 </div>

puppetboard/templates/_macros.html

@@ -5,7 +5,11 @@
 <table class="filter-table table table-striped {% if condensed %}table-condensed{% endif%}" style="table-layout:fixed">
   <thead>
     <tr>
+      {% if show_node %}
+      <th>Node</th>
+      {% else %}
       <th>Fact</th>
+      {% endif %}
       <th>Value</th>
     </tr>
   </thead>
@@ -13,9 +17,9 @@
     {% for fact in facts %}
     <tr>
       {% if show_node %}
-        <td>{{fact.node}}</td>
+        <td><a href="{{url_for('node', node_name=fact.node)}}">{{fact.node}}</a></td>
       {% else %}
-        <td>{{fact.name}}</td>
+        <td><a href="{{url_for('fact', fact=fact.name)}}">{{fact.name}}</a></td>
       {% endif %}
       <td style="word-wrap:break-word">{{fact.value}}</td>
     </tr>
@@ -23,6 +27,63 @@
   </tbody>
 </table>
 {%- endmacro %}
+{% macro facts_graph(facts, autofocus=False, condensed=False, show_node=False, margin_top=20, margin_bottom=20) -%}
+<script src="{{url_for('static', filename='js/Chart.js')}}"></script>
+<canvas id="factChart" width="300" height="300"></canvas>
+<script type="text/javascript">
+  var colors = ["#9B59B6", "#3498DB", "#2ECC71", "#1ABC9C", "#F1C40F", "#E67E22", "#E74C3C"];
+  var len_colors = colors.length;
+  var data = [
+{% for fact in facts|groupby('value') %}
+    {
+    label: "{{ fact.grouper }}",
+    value: {{ fact.list|length }}
+    },
+{% endfor %}
+    {
+    value: 0,
+    }
+  ]
+  for (var i = 0; i < data.length; i++) {
+    data[i].color = colors[i % len_colors];
+  }
+  var sorted_data = data.sort(function(a,b) { return parseFloat(b.value) - parseFloat(a.value)});
+  var top7 = sorted_data.slice(0,7);
+  var bottom = data.slice(7, -1);
+  var bottom_sum = 0;
+  for (var i = 0; i < bottom.length; i++) {
+    bottom_sum += bottom[i].value;
+  }
+  top7.push({ label: "Other", value: bottom_sum, color: "#B30202" });
+  var ctx = document.getElementById("factChart").getContext("2d");
+  new Chart(ctx).Pie(top7);
+</script>
+{%- endmacro %}
+{% macro facts_graph_value(facts, autofocus=False, condensed=False, show_node=False, margin_top=20, margin_bottom=20) -%}
+<script src="{{url_for('static', filename='js/Chart.js')}}"></script>
+<canvas id="factChart_value" width="300" height="300"></canvas>
+<script type="text/javascript">
+  var colors = ["#9B59B6", "#3498DB", "#2ECC71", "#1ABC9C", "#F1C40F", "#E67E22", "#E74C3C"];
+  var len_colors = colors.length;
+  var data = [
+{% for fact in facts|groupby('value') %}
+    {
+    label: "{{ fact.grouper }}",
+    value: {{ fact.list|length }}
+    },
+{% endfor %}
+    {
+    value: 0,
+    }
+  ]
+
+  for (var i = 0; i < data.length; i++) {
+    data[i].color = colors[i % len_colors];
+  }
+  var ctx = document.getElementById("factChart_value").getContext("2d");
+  new Chart(ctx).Pie(data.sort(function(a,b) { return parseInt(a.label) - parseInt(b.label)}));
+</script>
+{%- endmacro %}
 
 {% macro reports_table(reports, nodename, condensed=False, hash_truncate=False, show_conf_col=True, show_agent_col=True, show_host_col=True) -%}
 <div class="alert alert-info">

puppetboard/templates/fact.html

@@ -2,5 +2,7 @@
 {% import '_macros.html' as macros %}
 {% block content %}
 <h1>{{name}}</h1>
+{{macros.facts_graph(facts, autofocus=True, show_node=True, margin_bottom=10)}}
+{{macros.facts_graph_value(facts, autofocus=True, show_node=True, margin_bottom=10)}}
 {{macros.facts_table(facts, autofocus=True, show_node=True, margin_bottom=10)}}
 {% endblock content %}

puppetboard/templates/layout.html

@@ -25,6 +25,7 @@
               ('nodes', 'Nodes'),
               ('facts', 'Facts'),
               ('reports', 'Reports'),
+              ('metrics', 'Metrics'),
               ('query', 'Query'),
                 ] %}
                   <li{% if endpoint == request.endpoint %} class=active{% endif

puppetboard/templates/metric.html

@@ -0,0 +1,18 @@
+{% extends 'layout.html' %}
+{% block content %}
+<div class="page-header">
+  <h1>Metric
+    <small>{{name}}</small>
+  </h1>
+</div>
+<table class="table table-striped">
+  <tbody>
+    {% for key,value in metric %}
+    <tr>
+      <td>{{key}}</td>
+      <td>{{value}}</td>
+    </tr>
+    {% endfor %}
+  </tbody>
+</table>
+{% endblock content %}

puppetboard/templates/metrics.html

@@ -0,0 +1,9 @@
+{% extends 'layout.html' %}
+{% block content %}
+<h1>Metrics</h1>
+<ul>
+  {% for key,value in metrics %}
+  <li><a href="{{url_for('metric', metric=value)}}">{{key}}</li>
+  {% endfor %}
+</ul>
+{% endblock content %}

puppetboard/templates/node.html

@@ -27,7 +27,7 @@
       </tbody>
     </table>
   </div>
-  {% if config.PUPPETDB_EXPERIMENTAL %}
+  {% if config.PUPPETDB_API > 2 %}
   <div class="span4">
     <h1>Facts</h1>
     {{macros.facts_table(facts, condensed=True, margin_top=10)}}

puppetboard/templates/nodes.html

@@ -20,7 +20,7 @@
     <tr>
       <th>Hostname</th>
       <th>Catalog compiled at</th>
-      {% if config.PUPPETDB_EXPERIMENTAL %}
+      {% if config.PUPPETDB_API > 2 %}
       <th>Last report</th>
       <th>&nbsp;</th>
       {% endif %}
@@ -31,7 +31,7 @@
     <tr>
       <td><a href="{{url_for('node', node_name=node.name)}}">{{node.name}}</a></td>
       <td rel="utctimestamp">{{node.catalog_timestamp}}</td>
-      {% if config.PUPPETDB_EXPERIMENTAL %}
+      {% if config.PUPPETDB_API > 2 %}
       <td>
         {% if node.report_timestamp %}
         <span rel="utctimestamp">{{ node.report_timestamp }}</span>
@@ -40,7 +40,9 @@
         {% endif %}
       </td>
       <td>
+        {% if node.report_timestamp %}
         <a class="btn btn-small btn-primary" href="{{url_for('reports_node', node=node.name)}}">Reports</a>
+        {% endif %}
       </td>
       {% endif %}
     </tr>

puppetboard/templates/query.html

@@ -2,7 +2,7 @@
 {% block row_fluid %}
 <div class="span12">
   <div class="alert">
-    This is highly exeprimental and will likely set your server on fire.
+    This is highly experimental and will likely set your server on fire.
   </div>
 </div>
 <div class="container" style="margin-bottom:55px;">
@@ -23,7 +23,7 @@
         <div class="control-group {% if form.query.errors %} error {% endif %}">
           {{form.query.label(class_="control-label")}}
           <div class="controls">
-            {{form.query(class_="input-block-level", autofocus="autofocus", rows=5, placeholder="\"=\", \"name\" \"hostname\"")}}
+            {{form.query(class_="input-block-level", autofocus="autofocus", rows=5, placeholder="\"=\", \"name\", \"hostname\"")}}
             {% if form.query.errors %}
             <span class="help-inline">{% for error in form.query.errors %}{{error}}{% endfor %}</span>
             {% endif %}

puppetboard/utils.py

@@ -2,9 +2,9 @@ from __future__ import absolute_import
 from __future__ import unicode_literals
 
 from requests.exceptions import HTTPError, ConnectionError
-from pypuppetdb.errors import EmptyResponseError, ExperimentalDisabledError
+from pypuppetdb.errors import EmptyResponseError
 
-from flask import abort, flash
+from flask import abort
 
 def get_or_abort(func, *args, **kwargs):
     """Execute the function with its arguments and handle the possible
@@ -18,8 +18,6 @@ def get_or_abort(func, *args, **kwargs):
         abort(e.response.status_code)
     except ConnectionError:
         abort(500)
-    except ExperimentalDisabledError:
-        abort(412)
     except EmptyResponseError:
         abort(204)
 
@@ -40,22 +38,12 @@ def yield_or_stop(generator):
     generators and handle certain errors.
 
     Since this is also used in streaming responses where we can't just abort
-    a request we always yield empty and then raise StopIteration.
+    a request we raise StopIteration.
     """
     while True:
         try:
             yield next(generator)
         except StopIteration:
             raise
-        except ExperimentalDisabledError:
+        except (EmptyResponseError, ConnectionError, HTTPError):
-            yield
-            raise StopIteration
-        except EmptyResponseError:
-            yield
-            raise StopIteration
-        except ConnectionError:
-            yield
-            raise StopIteration
-        except HTTPError:
-            yield
             raise StopIteration

requirements.txt

@@ -1,3 +1,9 @@
 Flask==0.10.1
 Flask-WTF==0.8.4
-pypuppetdb=0.0.1
+Jinja2==2.7
+MarkupSafe==0.18
+WTForms==1.0.4
+Werkzeug==0.9.3
+itsdangerous==0.22
+pypuppetdb==0.0.4
+requests==1.2.3