Profile cover photo
Profile photo
João Pinto (Lamego)
Dad and IT Enthusiast
Dad and IT Enthusiast
About
Posts

Post has attachment
A picture is worth a thousand words, for a deeper understanding of a data pipeline.
Photo

Post has attachment
Testing the "file_delta" plugin is a bit tricky, since it requires writing/reading from the same file in a time coordinated pipeline.


https://github.com/mdatapipe/mdatapipe/blob/master/tests/plugins/collect/using/file_delta.yaml
Photo

Post has attachment
Sometimes we need a dynamic delta parsing of an acess log, this example provides a dynamic calc of rps / avg.


https://github.com/mdatapipe/mdatapipe/blob/master/examples/access_log_delta_stats.yaml
Photo

Building Python3.7 for CentOS / RHEL 5 is a painful experience, in case someone else has that need:


https://github.com/joaompinto/Python3_CentOS5


Post has attachment
If you like tactic games, you will love this one,
https://www.youtube.com/watch?v=G_itUeiNVhs

Post has attachment
A common issue of rapid application development is lack of organization, mdatapipe's core packages needed some love. Today they are much cleaner.

https://github.com/mdatapipe/mdatapipe
Photo

Post has attachment
Adding some more detailed documentation for plugins.
Photo

Post has attachment
Many plugins require additional python packages, you can now install all the plugins required for a data pipeline using: mdatapipe installdeps pipeline_file.
Photo

Post has attachment
Transform using field start plugin added. I love single screen plugins :)


https://github.com/mdatapipe/mdatapipe/blob/master/mdatapipe/plugins/transform/using/field_start.py
Photo

Post has attachment
Parsers can now be easily bench-marked, Grok vs CSV, CSV is a better option for access log parsing.
Photo
Wait while more posts are being loaded