Profile cover photo
Profile photo
Alvin Jin
About
Posts

Post has attachment
Install Tensorflow
$pip install virtualenv $mkdir envs Create a virtual environment $virtualenv ~/envs/tensorflow Activate the environment $source ~/envs/tensorflow/bin/activate Install Tensorflow (tensorflow)$pip install tensorflow​ Exit the virtual environment (tensorflow)$...
Add a comment...

Post has attachment
Kafka Partition Leader and Controller
Zookeeper and Controller: Every time a broker process starts, it registers itself with its id in Zookeper by creating an ephemeral node. When a broker loses connectivity to Zookeeper, the ephemeral node that the broker created when starting will be automati...
Add a comment...

Post has attachment
Why Kubernetes?
What is a Container? A container (Linux Container) at its core is an allocation, portioning, and assignment of host (compute) resources such as CPU Shares, Network I/O, Bandwidth, Block I/O, and Memory (RAM) so that kernel level constructs may jail-off, iso...
Add a comment...

Post has attachment
Kafka Consumer Challenges
Consumer Retry 1. When you encounter a retriable error, is to commit the last record you processed successfully. Then, store the records that still need to be processed in a buffer, use the consumer pause() method to ensure that additional polls won't retur...
Add a comment...

Post has attachment
Kafka Connect
Kafka, being a streaming data platform, acts as a giant buffer that decouples the time-sensitivity requirements between producers and consumers. Kafka itself applies back-pressure on producers, and consumption rate is driven entirely by the consumers. If pr...
Add a comment...

Post has attachment
Avro Schema Registry
It’s useless to send the schema of the data along with the data each time (as we do with JSON). It’s not memory and network efficient. It’s smarter to just send an ID along the data that the other parties will use to understand how are encoded the data. On ...
Add a comment...

Post has attachment
Config Log4j to Send Kafka logs to Syslog
1. Configure Syslog Daemon for UDP Input $ sudo vim /etc/rsyslog.conf Uncomment these lines to accept UDP messages on the default port 514. $ModLoad imudp $UDPServerRun 514 local7.*      /var/log/kafka.log 2. Restart the rsyslog service so the changes take ...
Add a comment...

Post has attachment
Kafka Streams
Lambda vs Kappa Architecture A scalable high-latency batch system that can process historical data and a low-latency stream processing system that can't reprocess results. Use Kafka to retain the full log of the data you want to be able to reprocess Retaini...
Add a comment...

Post has attachment
Event Sourcing and CQRS
Event Sourcing enables building a forward-compatible application architecture - the ability to add more application in the future that need to process the same event but create a different materialized view. The essence of event sourcing: rather than perfor...
Add a comment...

Post has attachment
AWS Network and Security
VPC: Virtual Private Cloud enables you to launch AWS resources into a virtual network that you've defined. An internet Gateway allows your VPC to connect to the Internet. An Internet gateway is used to enable outbound access to the Internet from VPC the clu...
Add a comment...
Wait while more posts are being loaded