720+% Returns in 3 years on Cryptocurrency using LSTM Neural Network Model and short listing Best Assets for Trading — VishvaAlgo Machine Learning Trading Bot

Unleashing the power of Neural Networks for creating Trading Bot for maximum profits.

Introduction:

Welcome to the world of algorithmic trading and machine learning, where innovation meets profitability. Over the past three years, I’ve dedicated myself to developing algorithmic trading systems that harness the power of various strategies. Through relentless experimentation and refinement, I’ve achieved impressive returns across multiple strategies, delighting members of my Patreon community with consistent profits.

In the pursuit of excellence, I recently launched VishvaAlgo, a machine learning-based algorithmic trading system that leverages neural network classification models. This cutting-edge platform has already demonstrated remarkable results, delivering exceptional returns to traders in the cryptocurrency market. Through a series of articles and practical demonstrations, I’ve shared insights on transitioning from traditional algorithmic trading to deploying practical machine learning models, showcasing their effectiveness in real-world trading environments.

In this article, we delve into the trans-formative potential of algorithmic trading and machine learning, focusing on the effectiveness of neural networks, specifically the LSTM technique. Building upon our past successes, we set out to demonstrate the remarkable profitability achievable with advanced machine learning models, using Bitcoin (BTC) and Ethereum (ETH) as our primary assets.

Our analysis focuses on Ethereum pricing in USDT, utilizing 15-minute candlestick data spanning from January 1st, 2021, to October 22nd, 2023, comprising over 97,000 rows of data and more than 190 features. By leveraging neural network models for prediction, we aim to identify optimal long and short positions, showcasing the potential of deep learning in financial markets.

Our story is one of relentless innovation, fueled by a burning desire to unlock the full potential of Deep Learning in the pursuit of profit. In this article, we invite you to join us as we unravel the exciting tale of our transformation from humble beginnings to groundbreaking success.

LSTM classification time series model for crypto

Our Algorithmic Trading Vs/+ Machine Learning Vs/+ Deep Learning Journey so far?

Stage 1:

We have developed a crypto Algorithmic Strategy which gave us huge profits when ran on multiple crypto assets (138+) with a profit range of 8787%+ in span of 3 years (almost).

“The 8787%+ ROI Algo Strategy Unveiled for Crypto Futures! Revolutionized With Famous RSI, MACD, Bollinger Bands, ADX, EMA” — Link

We have run live trading in dry-run mode for the same for 7 days and details about the same have been shared in another article.

“Freqtrade Revealed: 7-Day Journey in Algorithmic Trading for Crypto Futures Market” — Link

After successful backtest results and forward testing (live trading in dry-run mode), we planned to improve the odds of making more profit for the same. (To lower stop-losses, increase odds of winning more , reduce risk factor and other important things)

Stage 2:

We have worked on developing a strategy alone without freqtrade setup (avoiding trailing stop loss, multiple asst parallel running, higher risk management setups that freqtrade provides for free (it is a free open source platform) and then tested it in market, then optimized it using hyper parameters and then , we got some +ve profits from the strategy

“How I achieved 3000+% Profit in Backtesting for Various Algorithmic Trading Bots and how you can do the same for your Trading Strategies — Using Python Code” — Link

Stage 3:

As we have tested our strategy only on 1 Asset , i.e; BTC/USDT in crypto market, we wanted to know if we can segregate the whole collective assets we have (Which we have used for developing Freqtrade Strategy earlier) segregate them into different clusters based on their volatility, it becomes easy to do trading for certain volatile assets and won’t hit huge stop-losses for others if worked on implementing based on coin volatility.

We used K-nearest Neighbors (KNN Means) to identify different clusters of assets out of 138 crypto assets we use in our freqtrade strategy, which gave us 8000+% profits during backtest.

“Hyper Optimized Algorithmic Strategy Vs/+ Machine Learning Models Part -1 (K-Nearest Neighbors)” — Link

Stage 4:

Now, we want to introduce Unsupervised Machine Learning model — Hidden Markov Model (HMMs) to identify trends in the market and trade during only profitable trends and avoid sudden pumps, dumps in market, avoid negative trends in market. Below explanation unravels the same.

“Hyper Optimized Algorithmic Strategy Vs/+ Machine Learning Models Part -2 (Hidden Markov Model — HMM)” — Link

Stage 5:

I worked on using XGBoost Classifier to identify long and short trades using our old signal. Before using it, we ensured that the signal algorithm we had previously developed was hyper-optimized. Additionally, we introduced different stop-loss and take-profit parameters for this setup, causing the target values to change accordingly. We also adjusted the parameters used for obtaining profitable trades based on the stop-loss and take-profit values. Later, we tested the basic XGBClassifier setup and then enhanced the results by adding re-sampling methods. Our target classes, which include 0’s (neutral), 1’s (for long trades), and 2’s (for short trades), were imbalanced due to the trade execution timing. To address this imbalance, we employed re-sampling methods and performed hyper-optimization of the classifier model. Subsequently, we evaluated if the model performed better with other classifier models such as SVC, CatBoost, and LightGBM, in combination with LSTM and XGBoost. Finally, we concluded by analyzing the results and determining feature importance parameters to identify the most productive features.

“Hyper Optimized Algorithmic Strategy Vs/+ Machine Learning Models Part -3 (XGBoost Classifier , LGBM Classifier, CatBoost Classifier, SVC, LSTM with XGB and Multi level Hyper-optimization)” — Link

Stage 6:

In that stage, I utilized the CatBoostClassifier along with resampling and sample weights. I incorporated multiple time frame indicators such as volume, momentum, trend, and volatility into my model. After running the model, I performed ensembling techniques to enhance its overall performance. The results of my analysis showed a significant increase in profit from 54% to over 4600% during backtesting. Additionally, I highlighted the impressive performance metrics including recall, precision, accuracy, and F1 score, all exceeding 80% for each of the three trading classes (0 for neutral, 1 for long, and 2 for short trades).

“From 54% to a Staggering 4648%: Catapulting Cryptocurrency Trading with CatBoost Classifier, Machine Learning Model at Its Best” — Link

Stage 7:

In this stage, the ensemble method combining TCN and LSTM neural network models has demonstrated exceptional performance across various datasets, outperforming individual models and even surpassing buy and hold strategies. This underscores the effectiveness of ensemble learning in improving prediction accuracy and robustness.

“Bitcoin/BTC 4750%+ , Etherium/ETH 11,270%+ profit in 1023 days using Neural Networks, Algorithmic Trading Vs/+ Machine Learning Models Vs/+ Deep Learning Model Part — 4 (TCN, LSTM, Transformer with Ensemble Method)” — Link

Stage 8:

Experience the future of trading with VishvaAlgo v3.8. With its advanced features, unparalleled risk management capabilities, and ease of integration of ML and neural network models, VishvaAlgo is the ultimate choice for traders seeking consistent profits and peace of mind. Don’t miss out on this opportunity to revolutionize your trading journey.

Purchase Link: VishvaAlgo V3.8 Live Crypto Trading Using Machine Learning Model

“VishvaAlgo v3.0 — Revolutionize Your Live Cryptocurrency Trading system Enhanced with Machine Learning (Neural Network) Model. Live Profits Screenshots Shared” — Link

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

The code Explanation:

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

# Remove Future Warnings
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

# Suppress PerformanceWarning
warnings.filterwarnings("ignore")
# General
import numpy as np
# Data Management
import pandas as pd
# Machine Learning
from catboost import CatBoostClassifier
from sklearn.model_selection import train_test_split
from sklearn.model_selection import RandomizedSearchCV, cross_val_score
from sklearn.model_selection import RepeatedStratifiedKFold
from sklearn.linear_model import LogisticRegression
# ensemble
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.ensemble import StackingClassifier
from sklearn.ensemble import VotingClassifier
#Sampling Methods
from imblearn.over_sampling import ADASYN
#Scaling
from sklearn.preprocessing import MinMaxScaler
# Binary Classification Specific Metrics
from sklearn.metrics import RocCurveDisplay as plot_roc_curve
# General Metrics
from sklearn.metrics import accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import confusion_matrix, classification_report, roc_curve, roc_auc_score, accuracy_score
from sklearn.metrics import precision_score
from sklearn.metrics import ConfusionMatrixDisplay

# Reporting
import matplotlib.pyplot as plt
from matplotlib.pylab import rcParams
from xgboost import plot_tree
#Backtesting
from backtesting import Backtest
from backtesting import Strategy
#hyperopt
from hyperopt import fmin, tpe, hp
from pandas_datareader.data import DataReader
import json
from datetime import datetime
import talib as ta
import ccxt
# from sklearn.model_selection import train_test_split
from sklearn.utils import class_weight
from keras.models import Sequential
from keras.layers import LSTM, Dense, Dropout
from keras.optimizers import Adam
# from keras.wrappers.scikit_learn import KerasClassifier
from sklearn.ensemble import VotingClassifier
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

Import Statements (Lines 1–18):

• Warnings (Lines 1–4):
• These lines suppress warnings that might appear during execution. While this can be helpful for uninterrupted training, it’s generally recommended to address the warnings themselves for better debugging and understanding potential issues.
• General Libraries (Lines 5–7):
• numpy (np): Provides numerical computing capabilities, often used for array operations and mathematical functions. Not directly applicable to article writing.
• pandas (pd): Used for data manipulation, analysis, and visualization. Essential for working with structured data in articles (e.g., tables, charts).
• Machine Learning Libraries (Lines 8–13):
• catboost (not explicitly imported here): Provides a powerful gradient boosting library for machine learning tasks. Not directly relevant to article writing unless you're discussing specific machine learning algorithms.
• scikit-learn (various submodules): A comprehensive machine learning library. Parts might be useful for illustrating concepts or comparing approaches in articles:
• train_test_split: Splits data into training and testing sets for model evaluation.
• RandomizedSearchCV, cross_val_score, RepeatedStratifiedKFold: Techniques for hyperparameter tuning and model evaluation (cross-validation).
• LogisticRegression: A linear classification model. Potentially relevant if discussing classification algorithms.
• Ensemble Methods (Lines 14–16):
• scikit-learn (submodules): Techniques for combining multiple models to improve performance. Not directly applicable to article writing.
• Sampling Methods (Line 17):
• imblearn: Provides tools for handling imbalanced datasets (where classes have unequal sizes). Not typically used in article writing itself.
• Scaling (Line 18):
• scikit-learn: Techniques for normalizing or standardizing data (often necessary for machine learning models). Can be relevant in articles to explain data preprocessing steps.

Metrics (Lines 19–33):

• Binary Classification Metrics (Lines 19–21):
• scikit-learn: Used to evaluate the performance of classification models, particularly for binary classification (two classes). Not directly applicable to article writing unless discussing model evaluation metrics.
• General Metrics (Lines 22–33):
• scikit-learn: Various metrics for evaluating model performance across different classification tasks. Can be useful in articles to explain how models are assessed:
• accuracy_score: Proportion of correct predictions.
• precision_score: Proportion of true positives among predicted positives.
• confusion_matrix: Visualization of how many instances were classified correctly or incorrectly for each class.
• classification_report: Detailed report on model performance, including precision, recall, F1-score, and support for each class.
• roc_curve, roc_auc_score: Measures for assessing the Receiver Operating Characteristic (ROC) curve, which helps evaluate a model's ability to discriminate between classes.

Reporting (Lines 34–36):

• matplotlib.pyplot (plt): Used for creating visualizations like charts and graphs. Essential for presenting data and model results in articles.

Backtesting (Lines 37–38):

• backtesting: Library for backtesting trading strategies. Not relevant to article writing unless discussing financial applications of machine learning.

Hyperparameter Optimization (Lines 39–42):

• hyperopt: Library for hyperparameter tuning (finding the best settings for machine learning models). Not directly applicable to article writing.

Data Retrieval (Line 43):

• pandas_datareader: Facilitates data retrieval from various financial data sources. Not typically used in article writing itself.

Other Imports (Lines 44–50):

• json: For working with JSON data format (not directly used here).
• datetime: For working with date and time objects. Can be useful in articles for handling time-series data.
• talib: Technical analysis library for financial markets (not directly used here).
• ccxt (not explicitly imported here): Library for interacting with cryptocurrency exchanges (not relevant to article writing).

Context:

• Each library and module is imported with a specific purpose, such as data manipulation, machine learning, evaluation, visualization, backtesting, hyperparameter optimization, etc.
• These libraries and modules will be used throughout the code for various tasks like data preprocessing, model training, evaluation, optimization, and visualization.

# Define the path to your JSON file
file_path = './ETH_USDT_USDT-15m-futures.json'

# Open the file and read the data
with open(file_path, "r") as f:
    data = json.load(f)
df = pd.DataFrame(data)
# Extract the OHLC data (adjust column names as needed)
# ohlc_data = df[["date","open", "high", "low", "close", "volume"]]
df.rename(columns={0: "Date", 1: "Open", 2: "High",3: "Low", 4: "Adj Close", 5: "Volume"}, inplace=True)
# Convert timestamps to datetime objects
df["Date"] = pd.to_datetime(df['Date'] / 1000, unit='s')
df.set_index("Date", inplace=True)
# Format the date index
df.index = df.index.strftime("%m-%d-%Y %H:%M")
df['Close'] = df['Adj Close']
# print(df.dropna(), df.describe(), df.info())
data = df
data

To analyze historical cryptocurrency futures data, we can first load the data from a JSON file. The provided code demonstrates how to use Python’s json library to parse the JSON content into a dictionary. We then convert this dictionary into a pandas DataFrame for easier manipulation. The DataFrame is cleaned and transformed by renaming columns, converting timestamps to datetime objects, setting the date as the index, and formatting the date display for better readability.

Here’s the step-by-step explanation of the code:

1. Loading JSON Data:

• The code defines a file path (file_path) to a JSON file containing cryptocurrency data (presumably in the format of Open-High-Low-Close-Volume for Ethereum futures contracts traded with USDT).
• It opens the file for reading (with open(file_path, "r") as f:) and uses json.load(f) to parse the JSON content into a Python dictionary (data).

2. Converting to DataFrame:

• The code creates a pandas DataFrame (df) from the loaded dictionary (data). A DataFrame is a tabular data structure similar to a spreadsheet, making it easier to work with and analyze the data.

3. Data Cleaning and Transformation:

• This part assumes the JSON data has columns with numerical indices (0, 1, 2, etc.) instead of meaningful names. It renames these columns to more descriptive labels ("Date", "Open", "High", "Low", "Adj Close", "Volume") using df.rename(columns={...}, inplace=True).
• It converts the "Date" column from timestamps (likely in milliseconds since some epoch) to datetime objects using pd.to_datetime(). This makes it easier to work with dates and perform time-based operations.
• The code sets the "Date" column as the index of the DataFrame using df.set_index("Date", inplace=True). This allows you to efficiently access and filter data based on dates.
• It formats the date index using df.index.strftime("%m-%d-%Y %H:%M") to display dates in a more readable format (e.g., "05-14-2024 16:35").
• Finally, it assigns the column named "Adj Close" (assuming it represents the adjusted closing price) to a variable named "Close" for potentially clearer reference.

# Assuming you have a DataFrame named 'df' with columns 'Open', 'High', 'Low', 'Close', 'Adj Close', and 'Volume'
target_prediction_number = 2
time_periods = [6, 8, 10, 12, 14, 16, 18, 22, 26, 33, 44, 55]
name_periods = [6, 8, 10, 12, 14, 16, 18, 22, 26, 33, 44, 55]

df = data.copy()
new_columns = []
for period in time_periods:
    for nperiod in name_periods:
        df[f'ATR_{period}'] = ta.ATR(df['High'], df['Low'], df['Close'], timeperiod=period)
        df[f'EMA_{period}'] = ta.EMA(df['Close'], timeperiod=period*2)
        df[f'RSI_{period}'] = ta.RSI(df['Close'], timeperiod=period*0.5)
        df[f'VWAP_{period}'] = ta.SUM(df['Volume'] * (df['High'] + df['Low'] + df['Close']) / 3, timeperiod=period) / ta.SUM(df['Volume'], timeperiod=period)
        df[f'ROC_{period}'] = ta.ROC(df['Close'], timeperiod=period)
        df[f'KC_upper_{period}'] = ta.EMA(df['High'], timeperiod=period*2)
        df[f'KC_middle_{period}'] = ta.EMA(df['Low'], timeperiod=period*2)
        df[f'Donchian_upper_{period}'] = ta.MAX(df['High'], timeperiod=period)
        df[f'Donchian_lower_{period}'] = ta.MIN(df['Low'], timeperiod=period)
        macd, macd_signal, _ = ta.MACD(df['Close'], fastperiod=(period + 12), slowperiod=(period + 26), signalperiod=(period + 9))
        df[f'MACD_{period}'] = macd
        df[f'MACD_signal_{period}'] = macd_signal
        bb_upper, bb_middle, bb_lower = ta.BBANDS(df['Close'], timeperiod=period*0.5, nbdevup=2, nbdevdn=2)
        df[f'BB_upper_{period}'] = bb_upper
        df[f'BB_middle_{period}'] = bb_middle
        df[f'BB_lower_{period}'] = bb_lower
        df[f'EWO_{period}'] = ta.SMA(df['Close'], timeperiod=(period+5)) - ta.SMA(df['Close'], timeperiod=(period+35))
        
    
df["Returns"] = (df["Adj Close"] / df["Adj Close"].shift(target_prediction_number)) - 1
df["Range"] = (df["High"] / df["Low"]) - 1
df["Volatility"] = df['Returns'].rolling(window=target_prediction_number).std()
# Volume-Based Indicators
df['OBV'] = ta.OBV(df['Close'], df['Volume'])
df['ADL'] = ta.AD(df['High'], df['Low'], df['Close'], df['Volume'])

# Momentum-Based Indicators
df['Stoch_Oscillator'] = ta.STOCH(df['High'], df['Low'], df['Close'])[0]
# Calculate the Elliott Wave Oscillator (EWO)
#df['EWO'] = ta.SMA(df['Close'], timeperiod=5) - ta.SMA(df['Close'], timeperiod=35)
# Volatility-Based Indicators
# df['ATR'] = ta.ATR(df['High'], df['Low'], df['Close'], timeperiod=14)
# df['BB_upper'], df['BB_middle'], df['BB_lower'] = ta.BBANDS(df['Close'], timeperiod=20, nbdevup=2, nbdevdn=2)
# df['KC_upper'], df['KC_middle'] = ta.EMA(df['High'], timeperiod=20), ta.EMA(df['Low'], timeperiod=20)
# df['Donchian_upper'], df['Donchian_lower'] = ta.MAX(df['High'], timeperiod=20), ta.MIN(df['Low'], timeperiod=20)
# Trend-Based Indicators
# df['MA'] = ta.SMA(df['Close'], timeperiod=20)
# df['EMA'] = ta.EMA(df['Close'], timeperiod=20)
df['PSAR'] = ta.SAR(df['High'], df['Low'], acceleration=0.02, maximum=0.2)
# Set pandas option to display all columns
pd.set_option('display.max_columns', None)
# Displaying the calculated indicators
print(df.tail())
df.dropna(inplace=True)
print("Length: ", len(df))
df

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

This code demonstrates the calculation of various technical indicators using the talib library. The code iterates through different time periods to compute indicators like Average True Range (ATR), Exponential Moving Average (EMA), Relative Strength Index (RSI), and several others. Additionally, it calculates features like returns, range, and volatility to potentially use as input features for machine learning models.

1. Technical Indicator Calculations:

• The code iterates through two lists, time_periods and name_periods (which seem to have the same values here). This might be a placeholder for using different sets of periods for the indicators in the future.
• Within the loops, it calculates numerous technical indicators for each specified time period (period) using talib functions:
• Average True Range (ATR): Measures market volatility (df[f'ATR_{period}']).
• Exponential Moving Average (EMA): Calculates EMAs with a period twice the loop’s period (df[f'EMA_{period}']).
• Relative Strength Index (RSI): Calculates RSI with a period half the loop’s period (df[f'RSI_{period}']).
• Volume-Weighted Average Price (VWAP): Calculates VWAP for the period (df[f'VWAP_{period}']).
• Rate of Change (ROC): Calculates ROC for the period (df[f'ROC_{period}']).
• Keltner Channels (KC): Calculates upper and middle bands based on EMAs of highs and lows (df[f'KC_upper_{period}'], df[f'KC_middle_{period}']).
• Donchian Channels: Calculates upper and lower bands based on maximum and minimum highs/lows within the period (df[f'Donchian_upper_{period}'], df[f'Donchian_lower_{period}']).
• Moving Average Convergence Divergence (MACD): Calculates MACD and its signal line for the period (df[f'MACD_{period}'], df[f'MACD_signal_{period}']).
• Bollinger Bands (BB): Calculates upper, middle, and lower bands for the period (df[f'BB_upper_{period}'], df[f'BB_middle_{period}'], df[f'BB_lower_{period}']).
• Elliott Wave Oscillator (EWO): Calculates EWO for the period (df[f'EWO_{period}']).
• Target Prediction and Feature Engineering:
• The code defines a target_prediction_number (presumably the number of periods ahead you aim to predict).
• It calculates “Returns” as the percentage change in adjusted close prices over the target_prediction_number periods (df["Returns"]).
• It calculates “Range” as the difference between high and low prices divided by the low price (df["Range"]).
• It calculates “Volatility” as the rolling standard deviation of returns over the target_prediction_number periods (df["Volatility"]).
• Additional Indicators:
• The code calculates On-Balance Volume (OBV) and Accumulation Distribution Line (ADL) using talib functions (df['OBV'], df['ADL']).
• It calculates the Stochastic Oscillator using talib (df['Stoch_Oscillator']).
• It calculates the Parabolic Stop and Reversal (PSAR) using talib (df['PSAR']).

Data- Preprocessing — Setting up “Target” value for estimating future predictive values

# Target flexible way
pipdiff_percentage = 0.01  # 1% (0.01) of the asset's price for TP
SLTPRatio = 2.0  # pipdiff/Ratio gives SL
def mytarget(barsupfront, df1):
    length = len(df1)
    high = list(df1['High'])
    low = list(df1['Low'])
    close = list(df1['Close'])
    open_ = list(df1['Open'])  # Renamed 'open' to 'open_' to avoid conflict with Python's built-in function
    trendcat = [None] * length
    for line in range(0, length - barsupfront - 2):
        valueOpenLow = 0
        valueOpenHigh = 0
        for i in range(1, barsupfront + 2):
            value1 = open_[line + 1] - low[line + i]
            value2 = open_[line + 1] - high[line + i]
            valueOpenLow = max(value1, valueOpenLow)
            valueOpenHigh = min(value2, valueOpenHigh)
            if (valueOpenLow >= close[line + 1] * pipdiff_percentage) and (
                    -valueOpenHigh <= close[line + 1] * pipdiff_percentage / SLTPRatio):
                trendcat[line] = 2  # -1 downtrend
                break
            elif (valueOpenLow <= close[line + 1] * pipdiff_percentage / SLTPRatio) and (
                    -valueOpenHigh >= close[line + 1] * pipdiff_percentage):
                trendcat[line] = 1  # uptrend
                break
            else:
                trendcat[line] = 0  # no clear trend
return trendcat

This code defines a function mytarget that attempts to identify potential trends and set target values accordingly. It calculates the difference between the open price and upcoming highs/lows within a specified timeframe (barsupfront). Based on these differences and thresholds defined by pipdiff_percentage and SLTPRatio, the function classifies the trend as uptrend, downtrend, or no clear trend. These classifications could then be used to set target buy/sell prices in a trading strategy.

Here’s the breakdown of the code provided:

The provided code defines a function mytarget that aims to set target values (presumably for buying and selling) based on a trend classification. Here's a breakdown of its functionality:

Parameters:

• barsupfront (integer): The number of bars to look ahead from the current bar for trend classification.
• df1 (pandas DataFrame): The DataFrame containing OHLC (Open, High, Low, Close) prices.

Function Logic:

1. Initialization:

• It retrieves the length of the DataFrame (length).
• It extracts lists of high, low, close, and open prices (high, low, close, open_). Note that open is renamed to open_ to avoid conflicts with Python's built-in open function.
• It initializes a list trendcat with length elements, all set to None, which will eventually hold the trend category (uptrend, downtrend, or no trend) for each bar.

2. Trend Classification Loop:

• The code iterates through the DataFrame, starting from the barsupfront-th bar to the second-last bar (length - barsupfront - 2).
• Inside the loop:
• It calculates two values:
• valueOpenLow: Maximum difference between the open price at the current bar and the low prices in the next barsupfront + 1 bars.
• valueOpenHigh: Minimum difference between the open price at the current bar and the high prices in the next barsupfront + 1 bars.
• It checks these values against thresholds based on pipdiff_percentage (a percentage of the asset's price) and SLTPRatio:
• If valueOpenLow is greater than or equal to close[line + 1] * pipdiff_percentage (meaning the open is significantly lower than some of the upcoming lows) AND -valueOpenHigh is less than or equal to close[line + 1] * pipdiff_percentage / SLTPRatio (meaning the open is not significantly higher than some of the upcoming highs), it classifies the trend as downtrend (trendcat[line] is set to 2).
• Conversely, if valueOpenLow is less than or equal to close[line + 1] * pipdiff_percentage / SLTPRatio (meaning the open is significantly higher than some of the upcoming lows) AND -valueOpenHigh is greater than or equal to close[line + 1] * pipdiff_percentage (meaning the open is not significantly lower than some of the upcoming highs), it classifies the trend as uptrend (trendcat[line] is set to 1).
• If neither condition is met, it marks no clear trend (trendcat[line] remains 0).

3. Return:

• The function returns the trendcat list containing the trend classification for each bar (except the first barsupfront bars).
• pen_spark

#!!! pitfall one category high frequency
df['Target'] = mytarget(2, df)
df['Target'] = df['Target'].shift(1)
#df.tail(20)
df.replace([np.inf, -np.inf], np.nan, inplace=True)
df.dropna(axis=0, inplace=True)

# Convert columns to integer type
df = df.astype(int)
#df['Target'] = df['Target'].astype(int)
df['Target'].hist()
count_of_twos_target = df['Target'].value_counts().get(2, 0)
count_of_zeros_target = df['Target'].value_counts().get(0, 0)
count_of_ones_target = df['Target'].value_counts().get(1, 0)
percent_of_zeros_over_ones_and_twos = (100 - (count_of_zeros_target/ (count_of_zeros_target + count_of_ones_target + count_of_twos_target))*100)
print(f' count_of_zeros = {count_of_zeros_target}\n count_of_twos_target = {count_of_twos_target}\n count_of_ones_target={count_of_ones_target}\n percent_of_zeros_over_ones_and_twos = {round(percent_of_zeros_over_ones_and_twos,2)}%')

output of the above code

After assigning trend classifications (Target) based on the mytarget function, the code performs data cleaning by handling infinities and removing rows with missing values. It then analyzes the distribution of target values using a histogram and calculates the proportion of bars classified as each trend category. This helps assess the balance between clear uptrends, downtrends, and periods with no clear trend in the data.

1. Assigning Target Values and Shifting:

• The code assigns the output of mytarget(2, df) (presumably trend classifications) to the 'Target' column (df['Target'] = mytarget(2, df)).
• It then shifts the 'Target' values by one position upwards (df['Target'] = df['Target'].shift(1)) because the trend classification is based on future price movements. This means the target value for bar n is based on the trend classification for bar n-1.

2. Handling Infinities and Missing Values:

• The code replaces positive and negative infinity (np.inf and -np.inf) with NaN (Not a Number) values in the DataFrame (df.replace([np.inf, -np.inf], np.nan, inplace=True)). This is necessary because some mathematical operations cannot handle infinities.
• It then removes rows with missing values (NaN) from the DataFrame (df.dropna(axis=0, inplace=True)) to ensure clean data for further analysis.

3. Converting Data Types (Commented Out):

• The line df = df.astype(int) is commented out. This line would attempt to convert all columns in the DataFrame to integers. However, since the 'Target' column likely contains categorical values (1, 2, or 0), converting it to integer might not be meaningful. You'd typically only convert numerical columns to integers if necessary for calculations.

4. Analyzing Target Distribution:

• The code plots a histogram of the 'Target' column (df['Target'].hist()). This helps visualize the distribution of target values (uptrend, downtrend, or no trend) across the data.
• It then calculates the counts of each target value (1, 2, and 0) using value_counts().
• Finally, it calculates the percentage of bars classified as “no trend” relative to the sum of bars classified as uptrend and downtrend (percent_of_zeros_over_ones_and_twos). This provides insights into the balance between clear trends and unclear trends in the data.

This code segment effectively calculates target categories based on predefined criteria and provides insights into the distribution of these categories within the dataset.

Checking if the above Code is Giving Best Possible Returns for the “Target” Data Created:

# Check for NaN values:
has_nan = df['Target'].isnull().values.any()
print("NaN values present:", has_nan)

# Check for infinite values:
has_inf = df['Target'].isin([np.inf, -np.inf]).values.any()
print("Infinite values present:", has_inf)
# Count the number of NaN and infinite values:
nan_count = df['Target'].isnull().sum()
inf_count = (df['Target'] == np.inf).sum() + (df['Target'] == -np.inf).sum()
print("Number of NaN values:", nan_count)
print("Number of infinite values:", inf_count)
# Get the indices of NaN and infinite values:
nan_indices = df['Target'].index[df['Target'].isnull()]
inf_indices = df['Target'].index[df['Target'].isin([np.inf, -np.inf])]
print("Indices of NaN values:", nan_indices)
df['Target']
df = df.reset_index(inplace=False)
df['Date'] = pd.to_datetime(df['Date'])
df.set_index('Date', inplace=True)
def SIGNAL(df):
    return df['Target']
from backtesting import Strategy
class MyCandlesStrat(Strategy):  
    def init(self):
        super().init()
        self.signal1 = self.I(SIGNAL, self.data)
    
    def next(self):
        super().next() 
        if self.signal1 == 1:
            sl_pct = 0.025  # 2.5% stop-loss
            tp_pct = 0.025  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 - sl_pct)
            tp_price = self.data.Close[-1] * (1 + tp_pct)
            self.buy(sl=sl_price, tp=tp_price)
        elif self.signal1 == 2:
            sl_pct = 0.025  # 2.5% stop-loss
            tp_pct = 0.025  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 + sl_pct)
            tp_price = self.data.Close[-1] * (1 - tp_pct)
            self.sell(sl=sl_price, tp=tp_price)
            
bt = Backtest(df, MyCandlesStrat, cash=100000, commission=.001, exclusive_orders = True)
stat = bt.run()
stat

output of above code

1. Checking for Missing and Infinite Values:

• The code checks for the presence of NaN (Not a Number) and infinite values in the 'Target' column (df['Target']).
• It then counts the number of occurrences and retrieves the indices of these values.
• These checks are crucial because backtesting libraries typically cannot handle missing or infinite values in signals.

2. Backtesting Framework Setup:

• The code defines a function SIGNAL(df) that simply returns the 'Target' column values. This function essentially provides the buy/sell signals based on the target classifications (1 for uptrend buy, 2 for downtrend sell).
• It imports the Strategy class from the backtesting library.
• It defines a custom strategy class MyCandlesStrat that inherits from Strategy.
• The init method initializes an indicator named signal1 that holds the target values using the I function (presumably from backtesting).
• The next method defines the trading logic:
• If the signal1 is 1 (uptrend), it places a buy order with a stop-loss and take-profit based on percentages of the closing price.
• If the signal1 is 2 (downtrend), it places a sell order with a stop-loss and take-profit based on percentages of the closing price.

3. Backtesting and Evaluation:

• The code creates a Backtest object using the backtesting library. It provides the DataFrame (df), the strategy class (MyCandlesStrat), initial capital (cash), commission rate (commission), and sets exclusive_orders to True (potentially to prevent overlapping orders).
• It runs the backtest using the bt.run() method and stores the results in the stat variable.

Does this code definitively determine the effectiveness of the target values?

No, this code doesn’t definitively determine the effectiveness of the target values. Here’s why:

• Parameter Optimization: The stop-loss and take-profit percentages (sl_pct and tp_pct) are fixed in the code. Optimizing these parameters for the specific strategy and market conditions could potentially improve performance.
• Single Backtest Run: Running the backtest only once doesn’t account for the inherent randomness in financial markets. Ideally, you’d run the backtest multiple times with different random seeds to assess its robustness.

How to improve the code for target evaluation?

• Calculate Performance Metrics: Modify the code to calculate and print relevant performance metrics like Sharpe Ratio, drawdown, and total profit after the backtest run.
• Optimize Stop-Loss and Take-Profit: Implement a parameter optimization process to find the best stop-loss and take-profit values for the strategy using the target signals.
• Multiple Backtest Runs: Run the backtest with different random seeds (e.g., using a loop) and analyze the distribution of performance metrics to assess the strategy’s consistency.

By incorporating these improvements, wecan gain a more comprehensive understanding of how well the target values from the mytarget function perform in a backtesting framework. Remember, backtesting results are not guarantees of future performance, so real-world testing with a smaller capital allocation is essential before deploying a strategy with real money.

Scaling and splitting the dataframe for training and testing:

scaler = MinMaxScaler(feature_range=(0,1))

df_model = df.copy()
# Split into Learning (X) and Target (y) Data
X = df_model.iloc[:, : -1]
y = df_model.iloc[:, -1]
X_scaled = scaler.fit_transform(X)
# Define a function to reshape the data
def reshape_data(data, time_steps):
    samples = len(data) - time_steps + 1
    reshaped_data = np.zeros((samples, time_steps, data.shape[1]))
    for i in range(samples):
        reshaped_data[i] = data[i:i + time_steps]
    return reshaped_data
# Reshape the scaled X data
time_steps = 1  # Adjust the number of time steps as needed
X_reshaped = reshape_data(X_scaled, time_steps)
# Now X_reshaped has the desired three-dimensional shape: (samples, time_steps, features)
# Each sample contains scaled data for a specific time window
# Align y with X_reshaped by discarding excess target values
y_aligned = y[time_steps - 1:]  # Discard the first (time_steps - 1) target values
X = X_reshaped
y = y_aligned
print(len(X),len(y))
# Split data into train and test sets (considering time series data)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=False)

1. Data Preparation:

• Copying Data: It creates a copy of the original DataFrame (df_model = df.copy()) to avoid modifying the original data.

2. Splitting Features and Target:

• Separating Features (X) and Target (y): It separates the features (all columns except the last) and the target variable (the last column) using slicing (X = df_model.iloc[:, : -1], y = df_model.iloc[:, -1]).

3. Scaling Features:

• MinMaxScaler: It creates a MinMaxScaler object to scale the features between 0 and 1 (scaler = MinMaxScaler(feature_range=(0,1))). This can be helpful for some machine learning algorithms that work better with normalized data.
• Scaling X: It scales the feature data (X) using the fit_transform method of the scaler (X_scaled = scaler.fit_transform(X)).

4. Reshaping Data (Windowing):

• Reshape Function: It defines a function reshape_data that takes the data and the number of time steps (time_steps) as input.
• This function iterates through the data with a sliding window of time_steps and creates a new 3D array (reshaped_data).
• Each element in the new array represents a sample, containing a sequence of time_steps data points for each feature.
• Reshaping Scaled X: It defines the number of time steps (time_steps) and reshapes the scaled feature data (X_scaled) using the reshape_data function (X_reshaped = reshape_data(X_scaled, time_steps)).
• This step transforms the data into a format suitable for time series forecasting models that require sequences of past observations to predict future values.

5. Aligning Target with Reshaped Data:

• Discarding Excess Target Values: Since the reshaped data (X_reshaped) considers a window of time_steps, the corresponding target values need an adjustment. It discards the first time_steps - 1 target values from y to align with the reshaped data (y_aligned = y[time_steps - 1:]).

6. Final Splitting (Train-Test):

• Train-Test Split: It splits the reshaped features (X) and aligned target (y) into training and testing sets using train_test_split from scikit-learn (X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, shuffle=False)).
• It sets test_size=0.3 to allocate 30% of the data for testing and shuffle=False because shuffling data in time series can disrupt the temporal order.

Overall, this code effectively addresses key aspects of data preparation for time series forecasting models:

• Scaling features to a common range can improve model performance for some algorithms.
• Reshaping data into a 3D structure with time steps allows models to learn from sequences of past observations.
• Aligning the target variable with the reshaped data ensures the model predicts for the correct time steps.
• Splitting data into training and testing sets with shuffle=False preserves the temporal order for time series forecasting.

Additional Considerations:

• The choice of scaler (MinMaxScaler, StandardScaler, etc.) might depend on the specific model and data characteristics.
• You might explore different window sizes (time_steps) to see how they affect model performance.
• Techniques like stationarity checks and differencing might be necessary for certain time series data before applying these steps.

LSTM Model Manual Optimization

LSTM Classification time series model

from keras.layers import Input, Dense, Dropout
from keras.models import Model
from keras.optimizers import Adam
from keras.layers import LSTM
from keras.metrics import Precision, Recall
from keras.utils import to_categorical

class_weights = {0: 3.33, 1: 3.33, 2: 3.34}  # Adjust weights as needed


# Define LSTM model
def build_lstm_model(input_shape, units=193, dropout=0.2, lr=0.00002):
    inputs = Input(shape=input_shape)
    lstm_layer = LSTM(units=units, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(inputs)
    lstm_layer_2 = LSTM(units=128, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(lstm_layer)
    lstm_layer_3 = LSTM(units=96, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(lstm_layer_2)
    lstm_layer_4 = LSTM(units=48, dropout=0.2, recurrent_dropout=0.2)(lstm_layer_3)
    dense_layer = Dense(units=32, activation='relu')(lstm_layer_4)
    dropout_layer = Dropout(dropout)(dense_layer)
    outputs = Dense(3, activation='softmax')(dropout_layer)
    model = Model(inputs=inputs, outputs=outputs)
    optimizer = Adam(learning_rate=lr)
    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=[Precision(), 'accuracy', Recall()])
    return model

# Convert y_train to one-hot encoded format
y_train_one_hot = to_categorical(y_train, num_classes=3)

# Instantiate the model
model_lstm = build_lstm_model(input_shape=(X_train.shape[1], X_train.shape[2]))

# Fit the model to the training data
# model_lstm.fit(X_train, y_train_one_hot, epochs=150, batch_size=18, validation_split=0.2, verbose=1, class_weight=class_weights)
model_lstm.fit(X_train, y_train_one_hot, epochs=100, batch_size=18, validation_split=0.2, verbose=1)

This code builds and trains an LSTM (Long Short-Term Memory) neural network model to classify time series data into three categories: neutral (0), long (1), and short (2). The dataset comprises 100,000 rows of Ethereum (ETH) data with a 15-minute timeframe and 193 features.

Here’s a step-by-step explanation of the code:

Imports and Setup

from keras.layers import Input, Dense, Dropout, LSTM
from keras.models import Model
from keras.optimizers import Adam
from keras.metrics import Precision, Recall
from keras.utils import to_categorical

These lines import necessary components from Keras for building and training the LSTM model:

• Input: To define the input layer.
• Dense: For fully connected layers.
• Dropout: To prevent overfitting by randomly setting a fraction of input units to 0 at each update during training.
• LSTM: For LSTM layers, which are used to capture temporal dependencies in the data.
• Model: To create the model.
• Adam: Optimizer for updating model parameters.
• Precision, Recall: Metrics to evaluate model performance.
• to_categorical: To convert class vectors (integers) to binary class matrices.

Class Weight

class_weights = {0: 3.33, 1: 3.33, 2: 3.34}

These weights are used to handle class imbalance by giving more importance to underrepresented classes during training. Adjust these values as needed based on the distribution of your dataset.

Define the LSTM Model

def build_lstm_model(input_shape, units=193, dropout=0.2, lr=0.00002):
    inputs = Input(shape=input_shape)
    lstm_layer = LSTM(units=units, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(inputs)
    lstm_layer_2 = LSTM(units=128, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(lstm_layer)
    lstm_layer_3 = LSTM(units=96, return_sequences=True, dropout=0.2, recurrent_dropout=0.2)(lstm_layer_2)
    lstm_layer_4 = LSTM(units=48, dropout=0.2, recurrent_dropout=0.2)(lstm_layer_3)
    dense_layer = Dense(units=32, activation='relu')(lstm_layer_4)
    dropout_layer = Dropout(dropout)(dense_layer)
    outputs = Dense(3, activation='softmax')(dropout_layer)
    model = Model(inputs=inputs, outputs=outputs)
    optimizer = Adam(learning_rate=lr)
    model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=[Precision(), 'accuracy', Recall()])
    return model

This function builds the LSTM model:

1. Inputs: Defines the input shape based on the training data.
2. LSTM Layers: Four LSTM layers with decreasing units to capture sequential dependencies.

• return_sequences=True is used in the first three LSTM layers to return the full sequence of outputs.
• Dropout and recurrent dropout are used to prevent overfitting.

3. Dense Layer: A fully connected layer with ReLU activation.

4. Dropout Layer: Added after the dense layer to further reduce overfitting.

5. Output Layer: A dense layer with softmax activation to output probabilities for the three classes.

6. Compile Model: The model is compiled with the Adam optimizer, categorical cross-entropy loss, and metrics for precision, accuracy, and recall.

Convert Labels to One-Hot Encoding

y_train_one_hot = to_categorical(y_train, num_classes=3)

This line converts the target variable y_train into a one-hot encoded format, which is required for the categorical cross-entropy loss function.

Instantiate and Train the Model

model_lstm = build_lstm_model(input_shape=(X_train.shape[1], X_train.shape[2]))

model_lstm.fit(X_train, y_train_one_hot, epochs=100, batch_size=18, validation_split=0.2, verbose=1, class_weight=class_weights)

1. Instantiate the Model: The build_lstm_model function is called with the input shape derived from the training data.
2. Train the Model: The fit method trains the model on the training data.

• epochs=100: Number of epochs for training.
• batch_size=18: Number of samples per gradient update.
• validation_split=0.2: Fraction of the training data to be used as validation data.
• verbose=1: Verbosity mode (progress bar).
• class_weight=class_weights: To handle class imbalance during training.

Why LSTM is Better than 1D CNN for Time Series Data

1. Sequential Nature: LSTMs are specifically designed to handle sequential data and can capture temporal dependencies better than 1D CNNs, which are more suited for spatial data.
2. Memory Capability: LSTMs have a memory cell that can remember previous inputs, making them more effective for time series data where past values influence future values.
3. Handling Long Sequences: LSTMs can handle longer sequences due to their gating mechanisms, whereas 1D CNNs might struggle with long-term dependencies.

Summary

This LSTM model is tailored to classify ETH time series data into three categories: neutral, long, and short. The use of multiple LSTM layers helps capture complex temporal dependencies in the data, making it well-suited for this classification task. The model is trained with appropriate class weights to handle imbalance, and its performance is evaluated using precision, accuracy, and recall metrics.

from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

# # Reshape X_train and X_test back to their original shapes
# X_train_original_shape = X_train.reshape(X_train.shape[0], -1)
# X_test_original_shape = X_test.reshape(X_test.shape[0], -1)

# X_test_reshaped = X_test_original_shape.reshape(-1, 1, X_test_original_shape.shape[1])


# Now X_train_original_shape and X_test_original_shape have their original shapes

# Perform prediction on the original shape data
# y_pred = model.predict(X_test_reshaped)
y_pred = model_lstm.predict(X_test)


# Perform any necessary post-processing on y_pred if needed
# For example, if your model outputs probabilities, you might convert them to class labels using argmax:

y_pred_classes = np.argmax(y_pred, axis=1)

# Convert one-hot encoded y_test to class labels
y_test_classes = y_test

# Plot confusion matrix for test data
conf_matrix_test = confusion_matrix(y_test_classes, y_pred_classes)

# Plot confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(conf_matrix_test, annot=True, cmap='Blues', fmt='g', cbar=False)
plt.xlabel('Predicted labels')
plt.ylabel('True labels')
plt.title('Confusion Matrix - Test Data')
plt.show()

from sklearn.metrics import classification_report

# Generate classification report for test data
class_report = classification_report(y_test, y_pred_classes)

# Print classification report
print("Classification Report - Test Data:\n", class_report)

1. Imports:

• confusion_matrix from sklearn.metrics for calculating the confusion matrix.
• matplotlib.pyplot (plt) and seaborn (sns) for creating the confusion matrix visualization.
• classification_report from sklearn.metrics for generating a classification report.

2. Reshaping Data (Commented Out):

• The commented section addresses potential reshaping issues. It’s important to ensure your test data (X_test) has the correct shape expected by the model for prediction.

3. Prediction:

• y_pred = model_lstm.predict(X_test) performs predictions on the test data using your trained model.

4. Post-processing Predictions:

• y_pred_classes = np.argmax(y_pred, axis=2) assumes your model outputs probabilities for each class (neutral, long, short). This line converts the probabilities to class labels by using argmax (finding the index of the maximum value) along axis 2.

5. Converting True Labels:

• y_test_classes = y_test assumes your y_test data already contains class labels (0, 1, 2) for the test set.

6. Confusion Matrix:

• conf_matrix_test = confusion_matrix(y_test_classes, y_pred_classes) calculates the confusion matrix for the test data. It shows how many samples from each true class were predicted into each class by the model.

7. Visualization:

• The code creates a heatmap visualization of the confusion matrix using seaborn. This allows you to visually inspect how well the model classified each class. Ideally, you want to see high values on the diagonal, indicating correct classifications.

8. Classification Report:

• class_report = classification_report(y_test, y_pred_classes) generates a classification report for the test data. This report provides metrics like precision, recall, F1-score, and support for each class, offering a more detailed breakdown of the model's performance.
• pen_spark

Backtest with Test and Whole Data:

df_ens_test = df.copy() 

df_ens = df_ens_test[len(X_train):]

df_ens['lstm_neural_scaled'] =  np.argmax(model_lstm.predict(X_test), axis=1)

df_ens['lns'] = df_ens['lstm_neural_scaled'].shift(1).dropna().astype(int)

df_ens = df_ens.dropna()

df_ens['lns']

# df_ens = df.copy() 

# # df_ens = df_ens_test[len(X_train):]

# df_ens['lstm_neural_scaled'] =  np.argmax(model_lstm.predict(X), axis=1)

# df_ens['lns'] = df_ens['lstm_neural_scaled'].shift(-1).dropna().astype(int)

# df_ens = df_ens.dropna()

# df_ens['lns']

df_ens = df_ens.reset_index(inplace=False)
df_ens['Date'] = pd.to_datetime(df_ens['Date'])
df_ens.set_index('Date', inplace=True)

def SIGNAL_3(df_ens):
    return df_ens['lns']

class MyCandlesStrat_3(Strategy):  
    def init(self):
        super().init()
        self.signal1_1 = self.I(SIGNAL_3, self.data)
    
    def next(self):
        super().next() 
        if self.signal1_1 == 1:
            sl_pct = 0.055  # 10% stop-loss
            tp_pct = 0.055  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 - sl_pct)
            tp_price = self.data.Close[-1] * (1 + tp_pct)
            self.buy(sl=sl_price, tp=tp_price)
        elif self.signal1_1 == 2:
            sl_pct = 0.055  # 10% stop-loss
            tp_pct = 0.055  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 + sl_pct)
            tp_price = self.data.Close[-1] * (1 - tp_pct)
            self.sell(sl=sl_price, tp=tp_price)

            
bt_3 = Backtest(df_ens, MyCandlesStrat_3, cash=100000, commission=.001, exclusive_orders=False)
stat_3 = bt_3.run()
stat_3

backtest results for lstm classification time series model for crypto eth 15m timeframe 1000+ days result

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

This code takes the predictions from an LSTM model and uses them to generate trading signals for backtesting a trading strategy on Ethereum (ETH) time series data.

Data Preparation

df_ens_test = df.copy()
df_ens = df_ens_test[len(X_train):]

• df_ens_test: Creates a copy of the original DataFrame df.
• df_ens: Extracts a subset of df_ens_test starting from the length of the training data X_train. This assumes that X_train is the training dataset and the rest is for testing/validation.

df_ens['lstm_neural_scaled'] = np.argmax(model_lstm.predict(X_test), axis=1)

• lstm_neural_scaled: Uses the trained LSTM model to make predictions on the test data X_test. The np.argmax function converts the predicted probabilities into class labels (0, 1, or 2).

df_ens['lns'] = df_ens['lstm_neural_scaled'].shift(1).dropna().astype(int)
df_ens = df_ens.dropna()

• lns: Shifts the predictions by one time step to align the signals with the corresponding trading periods. dropna removes any rows with NaN values, ensuring that the shifted column aligns correctly.

df_ens = df_ens.reset_index(inplace=False)
df_ens['Date'] = pd.to_datetime(df_ens['Date'])
df_ens.set_index('Date', inplace=True)

• Reset and set index: Resets the index of the DataFrame and sets the Date column as the index. This ensures the DataFrame is correctly indexed by date.

Trading Signal and Strategy

def SIGNAL_3(df_ens):
    return df_ens['lns']

• SIGNAL_3: A function that returns the shifted predictions to be used as trading signals.

class MyCandlesStrat_3(Strategy):
    def init(self):
        super().init()
        self.signal1_1 = self.I(SIGNAL_3, self.data)
    def next(self):
        super().next()
        if self.signal1_1 == 1:
            sl_pct = 0.055
            tp_pct = 0.055
            sl_price = self.data.Close[-1] * (1 - sl_pct)
            tp_price = self.data.Close[-1] * (1 + tp_pct)
            self.buy(sl=sl_price, tp=tp_price)
        elif self.signal1_1 == 2:
            sl_pct = 0.055
            tp_pct = 0.055
            sl_price = self.data.Close[-1] * (1 + sl_pct)
            tp_price = self.data.Close[-1] * (1 - tp_pct)
            self.sell(sl=sl_price, tp=tp_price)

• MyCandlesStrat_3: A trading strategy class that implements the signals from SIGNAL_3.
• init: Initializes the strategy by storing the signal.
• next: Executes trades based on the signal. If the signal is 1 (long), it places a buy order with a stop-loss (SL) and take-profit (TP). If the signal is 2 (short), it places a sell order with SL and TP.

Backtesting

bt_3 = Backtest(df_ens, MyCandlesStrat_3, cash=100000, commission=.001, exclusive_orders=False)
stat_3 = bt_3.run()
stat_3

• Backtest: Runs the backtest using the Backtest class from the backtesting library.
• cash=100000: Initial capital.
• commission=.001: Trading commission.
• exclusive_orders=False: Allows overlapping orders.
• stat_3: Stores the backtest results.

Explanation of the Results

The backtest results provide various metrics that evaluate the performance of the trading strategy:

• Start: The start date of the backtest.
• End: The end date of the backtest.
• Duration: Total duration of the backtest period.
• Exposure Time [%]: Percentage of time the strategy was active in the market.
• Equity Final [$]: Final equity at the end of the backtest.
• Equity Peak [$]: Maximum equity value reached.
• Return [%]: Total return percentage.
• Buy & Hold Return [%]: Return if the asset was simply held during the period.
• Return (Ann.) [%]: Annualized return percentage.
• Volatility (Ann.) [%]: Annualized volatility percentage.
• Sharpe Ratio: Risk-adjusted return measure (higher is better).
• Sortino Ratio: Similar to Sharpe Ratio but penalizes downside volatility (higher is better).
• Calmar Ratio: Annualized return divided by the maximum drawdown (higher is better).
• Max. Drawdown [%]: Maximum observed loss from a peak.
• Avg. Drawdown [%]: Average drawdown during the period.
• Max. Drawdown Duration: Longest duration of a drawdown period.
• Avg. Drawdown Duration: Average duration of drawdown periods.
• # Trades: Number of trades executed.
• Win Rate [%]: Percentage of profitable trades.
• Best Trade [%]: Percentage return of the best trade.
• Worst Trade [%]: Percentage loss of the worst trade.
• Avg. Trade [%]: Average return per trade.
• Max. Trade Duration: Longest duration a trade was held.
• Avg. Trade Duration: Average duration trades were held.
• Profit Factor: Ratio of gross profits to gross losses.
• Expectancy [%]: Expected return per trade.
• SQN: System Quality Number, a measure of strategy performance.

Key Takeaways from the Results

1. Return and Equity: The strategy achieved a 41.35% return over the backtest period, ending with $141,347.90 in equity from an initial $100,000.
2. Volatility and Drawdowns: The strategy experienced significant volatility (72.45% annualized) with a maximum drawdown of -25.33%.
3. Performance Metrics: The Sharpe Ratio of 0.71 and Sortino Ratio of 1.76 indicate a decent risk-adjusted return, though not exceptional.
4. Trading Activity: The strategy executed 144 trades with a win rate of 48.61%. The average trade resulted in a slight loss (-0.22%).
5. Risk Management: The use of stop-loss and take-profit mechanisms helped manage risk but also resulted in mixed trade outcomes.

In summary, while the strategy achieved a positive return, it faced high volatility and drawdowns. Continuous optimization and risk management are necessary to improve its performance.

Save the Model:

from keras.models import save_model
import time
# save the lstm hyperopt model
model_lstm.save(f"model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_{round(stat_3['Sharpe Ratio'],2)}_time_{time.strftime('%Y%m%d%H%M%S')}.keras")

Explanation:

1. Import:

• save_model from keras.models is used to save the model.

Key Points:

• This approach provides a clear and informative way to save our model, including details about its training parameters, data, and performance.
• You can modify the filename structure to include additional information relevant to your needs.

Let’s Backtest entire data with saved model:

from keras.models import load_model

# # Load the ensemble_predict function using joblib
best_model = load_model('./model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras')

Intended Functionality:

1. Import:

• load_model from keras.models is used to load a saved model.

2. Loading the Model:

• best_model = load_model('./model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras'): This line attempts to load a model saved with the filename model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras from the directory ./models/.

df_ens = df.copy() 

# df_ens = df_ens_test[:len(X)]
y_pred = best_model.predict(X)

# Perform any necessary post-processing on y_pred if needed
# For example, if your model outputs probabilities, you might convert them to class labels using argmax:
# y_pred_classes = np.argmax(y_pred, axis=1)
y_pred = np.argmax(y_pred, axis=1) # for lstm, tcn, cnn models
# y_pred = np.argmax(y_pred, axis=2) # for transformers model
df_ens['best_model'] =  y_pred
df_ens['bm'] = df_ens['best_model'].shift(1).dropna().astype(int)
df_ens['ema_22'] = ta.EMA(df_ens['Close'], timeperiod=22)
df_ens['ema_55'] = ta.EMA(df_ens['Close'], timeperiod=55)
df_ens['ema_108'] = ta.EMA(df_ens['Close'], timeperiod=108)
df_ens = df_ens.dropna()
df_ens['bm']
df_ens = df_ens.reset_index(inplace=False)
df_ens['Date'] = pd.to_datetime(df_ens['Date'])
df_ens.set_index('Date', inplace=True)
def SIGNAL_010(df_ens):
    return df_ens['bm']
def SIGNAL_0122(df_ens):
    return df_ens['ema_22']
def SIGNAL_0155(df_ens):
    return df_ens['ema_55']
def SIGNAL_01108(df_ens):
    return df_ens['ema_108']
class MyCandlesStrat_010(Strategy):  
    def init(self):
        super().init()
        self.signal1_1 = self.I(SIGNAL_010, self.data)
        self.ema_1_22 = self.I(SIGNAL_0122, self.data)
        self.ema_1_55 = self.I(SIGNAL_0155, self.data)
        self.ema_1_108 = self.I(SIGNAL_01108, self.data)
    
    def next(self):
        super().next() 
        # if (self.signal1_1 == 1) and (self.data.Close > self.ema_1_22) and (self.ema_1_22 > self.ema_1_55) and (self.ema_1_55 > self.ema_1_108):
        #     sl_pct = 0.025  # 10% stop-loss
        #     tp_pct = 0.025  # 2.5% take-profit
        #     sl_price = self.data.Close[-1] * (1 - sl_pct)
        #     tp_price = self.data.Close[-1] * (1 + tp_pct)
        #     self.buy(sl=sl_price, tp=tp_price)
        # elif (self.signal1_1 == 2)  and (self.data.Close < self.ema_1_22) and (self.ema_1_22 < self.ema_1_55) and (self.ema_1_55 < self.ema_1_108):
        #     sl_pct = 0.025  # 10% stop-loss
        #     tp_pct = 0.025  # 2.5% take-profit
        #     sl_price = self.data.Close[-1] * (1 + sl_pct)
        #     tp_price = self.data.Close[-1] * (1 - tp_pct)
        #     self.sell(sl=sl_price, tp=tp_price)
            
    # def next(self):
    #     super().next() 
    #     if (self.signal1_1 == 1) and (self.ema_1_22 > self.ema_1_55) and (self.ema_1_55 > self.ema_1_108):
    #         sl_pct = 0.025  # 10% stop-loss
    #         tp_pct = 0.025  # 2.5% take-profit
    #         sl_price = self.data.Close[-1] * (1 - sl_pct)
    #         tp_price = self.data.Close[-1] * (1 + tp_pct)
    #         self.buy(sl=sl_price, tp=tp_price)
    #     elif (self.signal1_1 == 2) and (self.ema_1_22 < self.ema_1_55) and (self.ema_1_55 < self.ema_1_108):
    #         sl_pct = 0.025  # 10% stop-loss
    #         tp_pct = 0.025  # 2.5% take-profit
    #         sl_price = self.data.Close[-1] * (1 + sl_pct)
    #         tp_price = self.data.Close[-1] * (1 - tp_pct)
    #         self.sell(sl=sl_price, tp=tp_price)
            
        if (self.signal1_1 == 1):
            sl_pct = 0.035  # 10% stop-loss
            tp_pct = 0.025  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 - sl_pct)
            tp_price = self.data.Close[-1] * (1 + tp_pct)
            self.buy(sl=sl_price, tp=tp_price)
        elif (self.signal1_1 == 2):
            sl_pct = 0.035  # 10% stop-loss
            tp_pct = 0.025  # 2.5% take-profit
            sl_price = self.data.Close[-1] * (1 + sl_pct)
            tp_price = self.data.Close[-1] * (1 - tp_pct)
            self.sell(sl=sl_price, tp=tp_price)
            
bt_010 = Backtest(df_ens, MyCandlesStrat_010, cash=100000, commission=.001)
stat_010 = bt_010.run()
stat_010

720%+ returns for ETh in 1022 days using Neural Networks LSTM Model with VishvaAlgo

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

This code builds upon your previous strategy by incorporating a LSTM model prediction ('best_model') along with Exponential Moving Averages (EMAs) to generate buy and sell signals for a backtesting strategy. Here's a breakdown:

1. Data Preparation:

• df_ens = df.copy(): Creates a copy of the original DataFrame (df).
• y_pred = best_model.predict(X): Makes predictions on the entire DataFrame (X) using your loaded LSTM model (best_model).
• df_ens['best_model'] = y_pred: Adds a new column 'best_model' to the DataFrame containing the model predictions.
• df_ens['bm'] = df_ens['best_model'].shift(1).dropna().astype(int): Similar to before, this creates a shifted signal column 'bm' based on the predicted labels, but here it might include predictions for the entire DataFrame.
• df_ens['ema_22'] = ta.EMA(df_ens['Close'], timeperiod=22): Calculates the 22-period EMA for the 'Close' price and adds it as a new column 'ema_22'.
• df_ens['ema_55'] = ta.EMA(df_ens['Close'], timeperiod=55): Similar to above, calculates the 55-period EMA and adds it as 'ema_55'.
• df_ens['ema_108'] = ta.EMA(df_ens['Close'], timeperiod=108): Calculates the 108-period EMA and adds it as 'ema_108'.
• df_ens = df_ens.dropna(): Removes rows with missing values (likely the first row due to shifting).

2. Signal Functions (Outside the Code Block):

• These functions (SIGNAL_010, SIGNAL_0122, etc.) simply return the corresponding columns from the DataFrame ('bm', 'ema_22', etc.) used for generating the signals.

3. Backtesting Strategy Class (MyCandlesStrat_010):

• Inherits from Strategy.
• def init(self): Initializes indicators for the LSTM model predictions (self.signal1_1) and EMAs (self.ema_1_22, etc.).

4. Backtesting Logic (in next function):

• The commented-out section shows a more complex logic considering the relationship between the LSTM predictions and the EMAs for buy/sell decisions.
• The current active section uses a simpler approach:
• If self.signal1_1 (LSTM prediction) is 1 (long):
• Buy with stop-loss (SL) at 3.5% below current close and take-profit (TP) at 2.5% above.
• If self.signal1_1 is 2 (short):
• Sell with SL at 3.5% above current close and TP at 2.5% below.

5. Backtesting and Results:

• bt_010 = Backtest(df_ens, MyCandlesStrat_010, cash=100000, commission=.001): Creates a backtest object using the DataFrame, strategy class, and other parameters.
• stat_010 = bt_010.run(): Runs the backtest and stores the results in stat_010.
• stat_010: This variable likely contains the backtesting statistics you can analyze.

Explanation of the Updated Results

The updated backtest results provide various metrics that evaluate the performance of the trading strategy over the specified period:

• Start: 2021–01–06 05:30:00
• End: 2023–10–22 15:30:00
• Duration: 1019 days 10:00:00
• Exposure Time [%]: 77.46% — The percentage of time the strategy was actively holding positions in the market.
• Equity Final [$]: $820,380.357 — The final equity at the end of the backtest.
• Equity Peak [$]: $1,198,316.882 — The highest equity value reached during the backtest.
• Return [%]: 720.38% — The total return percentage over the backtest period.
• Buy & Hold Return [%]: 47.19% — The return if the asset was simply held during the period.
• Return (Ann.) [%]: 110.58% — The annualized return percentage.
• Volatility (Ann.) [%]: 166.00% — The annualized volatility percentage.
• Sharpe Ratio: 0.67 — The risk-adjusted return measure (higher is better).
• Sortino Ratio: 2.72 — Similar to Sharpe Ratio but penalizes downside volatility (higher is better).
• Calmar Ratio: 2.37 — Annualized return divided by the maximum drawdown (higher is better).
• Max. Drawdown [%]: -46.70% — The maximum observed loss from a peak.
• Avg. Drawdown [%]: -4.32% — The average drawdown during the period.
• Max. Drawdown Duration: 277 days 21:00:00 — The longest duration of a drawdown period.
• Avg. Drawdown Duration: 5 days 16:57:00 — The average duration of drawdown periods.
• # Trades: 1910 — The number of trades executed.
• Win Rate [%]: 63.66% — The percentage of profitable trades.
• Best Trade [%]: 2.49% — The percentage return of the best trade.
• Worst Trade [%]: -3.69% — The percentage loss of the worst trade.
• Avg. Trade [%]: 0.19% — The average return per trade.
• Max. Trade Duration: 15 days 21:30:00 — The longest duration a trade was held.
• Avg. Trade Duration: 0 days 13:33:00 — The average duration trades were held.
• Profit Factor: 1.18 — The ratio of gross profits to gross losses.
• Expectancy [%]: 0.23% — The expected return per trade.
• SQN: 1.24 — System Quality Number, a measure of strategy performance.

Key Takeaways from the Updated Results

1. High Return: The strategy achieved a 720.38% return over the backtest period, significantly outperforming the buy-and-hold return of 47.19%.
2. Volatility and Drawdowns: The strategy experienced high volatility (166% annualized) and a substantial maximum drawdown of -46.70%. However, it managed to recover and end with a significant profit.
3. Performance Metrics: The Sharpe Ratio of 0.67 and Sortino Ratio of 2.72 indicate a decent risk-adjusted return, with the Sortino Ratio showing better performance when focusing on downside risk.
4. Trading Activity: The strategy executed a large number of trades (1910) with a win rate of 63.66%. The average trade resulted in a small but positive return (0.19%).
5. Risk Management: The use of stop-loss and take-profit mechanisms helped manage risk and contributed to a positive overall expectancy (0.23%).

In summary, the strategy demonstrated strong performance with a high return, but it also encountered significant volatility and drawdowns. Continuous optimization and risk management are essential to maintain and improve its performance.

Conclusion for LSTM Time Series Classification Model

The LSTM time series classification model demonstrated effective performance in classifying Ethereum price movements into neutral (0), long (1), and short (2) categories. Below are the key takeaways and conclusions from the implementation and backtesting of the model:

Key Takeaways

1. Model Architecture:

• The LSTM model was constructed with four layers of LSTM units, each with dropout and recurrent dropout for regularization.
• Dense layers with ReLU activation and dropout were added to capture complex patterns in the data.
• The model was compiled using the Adam optimizer and trained with categorical cross-entropy loss, precision, accuracy, and recall as metrics

2. Class Balancing:

• Class weights were applied to address the imbalance in the dataset, ensuring that each class was given appropriate importance during training.

3. Prediction and Signal Generation:

• The model’s predictions on the test dataset were shifted to align the signals with the corresponding trading periods.
• Trading signals were generated based on the model’s predictions and used to drive a trading strategy.

4. Trading Strategy:

• The trading strategy used the LSTM model’s predictions to execute trades with defined stop-loss and take-profit levels.
• The strategy was backtested on a substantial dataset, and various performance metrics were recorded.

Performance Summary

• Total Return: The strategy achieved a total return of 720.38%, significantly outperforming the buy-and-hold return of 47.19%.
• Annualized Return: The annualized return was 110.58%, indicating strong performance over the test period.
• Volatility: The strategy experienced high annualized volatility at 166.00%, reflecting the inherent risks of cryptocurrency trading.
• Drawdowns: The maximum drawdown was -46.70%, with an average drawdown of -4.32%. Managing these drawdowns is crucial for long-term success.
• Sharpe Ratio: The Sharpe Ratio of 0.67 indicates a reasonable risk-adjusted return.
• Sortino Ratio: The Sortino Ratio of 2.72 shows better performance when focusing on downside risk.
• Trade Metrics: With 1910 trades executed, the strategy had a win rate of 63.66% and an average trade duration of 0 days 13:33:00. The profit factor was 1.18, indicating a positive overall outcome.

Conclusion

The LSTM time series classification model has proven to be a valuable tool for predicting Ethereum price movements and generating profitable trading signals. Despite the high volatility and substantial drawdowns, the model’s robust returns and positive expectancy demonstrate its potential in algorithmic trading.

However, there are areas for further improvement and optimization:

1. Risk Management: Implementing advanced risk management techniques could help mitigate drawdowns and volatility.
2. Model Optimization: Continuous refinement of the LSTM model’s architecture and hyperparameters can enhance performance.
3. Broader Application: Extending the model to other assets and timeframes could provide additional insights and opportunities.
4. Live Trading: Testing the strategy in a live trading environment would provide practical insights and validate its real-world applicability.

Overall, the LSTM model’s ability to capture complex patterns in time series data makes it a powerful tool for trading strategies, with significant potential for generating high returns.

Applying neural network LSTM Model for Other Assets and Short List the Best:

From here on we will explain about how to use the same trained model to short list best assets after doing certain backtest on all the assets after downloading the data from tradingview for backtest

Importing Necessary packages and setting up Model & Exchange APi with CCXT

import time
import logging
import io
import contextlib
import glob
import ccxt
from datetime import datetime, timedelta, timezone
import keras
from keras.models import save_model, load_model
import numpy as np
import pandas as pd
import talib as ta
from sklearn.preprocessing import MinMaxScaler
import warnings
from threading import Thread, Event
import decimal
import joblib
from tcn import TCN

# from pandas.core.computation import PerformanceWarning
# Suppress PerformanceWarning
warnings.filterwarnings("ignore")
# NOTE: Train your own model from the other notebook I have shared and use the most successful trained model here.
# model_file_path = './model_lstm_1tp_1sl_2p5SlTp_April_5th_ShRa_1_49_15m.hdf5'
model_file_path = './model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras'
model_name = model_file_path.split('/')[-1]
##################################### TO Load A Model #######################################
# NOTE: for LSTM based neural network model  you can directly load_model with model_file_path as given below
# Load your pre-trained model, keras trained model will only take load_model from keras.models and not from joblib
model = load_model(model_file_path)
# # or
# model = tf.keras.models.load_model(model_file_path)
# NOTE: for TCN based neural network model, you need to add custom_objects while loading the model, it is given below
# # Define a dictionary to specify custom objects
# custom_objects = {'TCN': TCN}
# model = load_model(model_file_path, custom_objects = custom_objects)

##########################################################################################
########################## Adding the exchange information ##############################
exchange = ccxt.binanceusdm(
    {
        'enableRateLimit': True,  # required by the Manual
        # Add any other authentication parameters if needed
        'rateLimit': 250, 'verbose': True
    }
    )
# NOTE: I used https://testnet.binancefuture.com/en/futures/BTCUSDT for testnet API (this has very bad liquidity issue for various assets and many other issues but can be used for purely testiug purpose)
#  kraken testnet creds pubkey - K9dS2SK8JURMl9F300lguUhOS/ao3HM+tfRMgJGed+JhDfpJhvsC/y           privatekey - /J/03PPyPwsrPsKZYtLqOQNPLKZJattT6i15Bpg14/6ALokHHY/MBb1p6tYKyFgkKXIJIOMbBsFRfL3aBZUvQ1
# api_key = '8f7080f8821b58a53f5c49f00cbff7fdcce1cca9c9154ea'
# secret_key = '1e58391a46a7dbb098aa5121d3e69e3a6660ba8c38f'

# exchange.apiKey = api_key
# exchange.secret = secret_key
# exchange.set_sandbox_mode(True)

# NOTE: if u want to go live, un commenb below 5 lines and comment 5 lines above and change to your own api_key and secret_key (below one ius a dummy and also make sure to give "futres" permission while creating your api in the exchange)
api_key = 'CxUdC80c3Y5Nf1iRJMZJelOCfFJWISbQsasPraCb4Zdskx7MM8uCl'
secret_key = 'p4XwsZwmmNswzDHzE5TSUOgXT5tASArfSO0pxfYrBMtezlCpDGtz'
exchange.apiKey = api_key
exchange.secret = secret_key
exchange.set_sandbox_mode(False)
#######################################################################################
    # exchange.set_sandbox_mode(True)
exchange.has
# exchange.fetchBalance()["info"]["assets"]
exchange.options = {'defaultType': 'future', # or 'margin' or 'spot'
                    'timeDifference': 0,  # Set an appropriate initial value for time difference
                        'adjustForTimeDifference': True,
                        'newOrderRespType': 'FULL',
                        'defaultTimeInForce': 'GTC'}

The provided code snippet demonstrates how to load our trained model and connect to a cryptocurrency exchange (Binance) for potential shortlisting of assets based on backtesting. Here’s a breakdown:

Imports:

• Standard libraries for time, logging, data manipulation (pandas, numpy), machine learning (Keras, scikit-learn), technical indicators (talib), threading, and others.

Model Loading:

• Comments explain the difference in loading a model based on its type:
• LSTM Model: Uses load_model from keras.models directly (as shown in your code).
• TCN Model: Requires specifying custom objects (custom_objects={'TCN': TCN}) during loading.

Exchange Connection:

• Creates a ccxt.binanceusdm object (exchange) to interact with the Binance exchange.
• Sets API credentials and enables rate limiting for responsible API usage.
• Comments mention testnet and live API usage options.

Important Notes:

• Replace API Keys: Replace the dummy api_key and secret_key with your actual Binance API credentials (if going live). Ensure your API has "futures" permission.
• Backtesting Not Shown: This code focuses on model loading and exchange connection. The actual backtesting loop and asset shortlisting logic are not included.

Next Steps:

1. Backtesting Loop: You’ll need to implement a loop to iterate through your desired assets:

• Download historical data from the exchange (using exchange.fetch_ohlcv) for each asset.
• Preprocess the data (scaling, feature engineering).
• Make predictions using your loaded model (model.predict).
• Apply your backtesting strategy (similar to previous examples) incorporating predictions and potentially technical indicators.
• Store backtesting results for each asset.

1. Shortlisting: Analyze the stored backtesting results and apply filters/sorting based on your chosen metrics to shortlist the best-performing assets.
2. Risk Management: Remember, backtesting is for evaluation, not a guarantee of future success. Implement proper risk management strategies before using these shortlisted assets in real trading.

from sklearn.preprocessing import MinMaxScaler
from backtesting import Strategy, Backtest
import os
import json
import pandas as pd
import talib as ta
import numpy as np
from concurrent.futures import ThreadPoolExecutor
import threading

import time
import ccxt
from keras.models import save_model, load_model
import warnings
import decimal
import joblib
import nest_asyncio
# from pandas.core.computation import PerformanceWarning
# Suppress PerformanceWarning
warnings.filterwarnings("ignore")
# Load your pre-trained model
# model = load_model('best_model_tcn_1sl_1tp_2p5SlTp_success.pkl')
# Define the custom_assets dictionary outside the loop
custom_assets = {}
# Function to load custom_assets from a text file
def load_custom_assets():
    if os.path.exists('custom_assets.txt'):
        try:
            with open('custom_assets.txt', 'r') as txt_file:
                return json.loads(txt_file.read())
        except json.JSONDecodeError as e:
            print(f"Error decoding JSON in custom_assets.txt: {e}")
            return {}
    else:
        print("custom_assets.txt file not found. Initializing an empty dictionary.")
        custom_assets = {}
        save_custom_assets(custom_assets)
        return custom_assets
# Define a threading lock
file_lock = threading.Lock()
# Function to save custom_assets to a text file
def save_custom_assets(custom_assets):
    with file_lock:
        with open('custom_assets.txt', 'w') as txt_file:
            json.dump(custom_assets, txt_file, indent=4)

The provided code focuses on managing custom assets and preparing for multi-threaded backtesting. Here’s a breakdown:

Imports:

• Includes libraries for data manipulation (pandas, numpy), technical indicators (talib), backtesting framework (backtesting), threading, and others.

Custom Assets Management:

custom_assets dictionary:

• Stores custom assets for backtesting (likely symbols or names).

load_custom_assets function:

• Checks for a file named custom_assets.txt.
• If the file exists, attempts to load the dictionary from the JSON content. Handles potential JSON decoding errors.
• If the file doesn’t exist, initializes an empty dictionary, saves it, and returns it.

save_custom_assets function:

• Uses a threading lock (file_lock) to ensure safe access to the file during potential concurrent writes.
• Saves the custom_assets dictionary as JSON to the custom_assets.txt file.

Next Steps:

1. Backtesting Function: You’ll likely define a function for the backtesting logic. This function would:

• Take an asset symbol as input.
• Download historical data for the asset.
• Preprocess the data (scaling, feature engineering).
• Make predictions using your loaded model.
• Apply your backtesting strategy (similar to previous examples) incorporating predictions and potentially technical indicators.
• Calculate and store backtesting results (Sharpe Ratio, drawdown, etc.) for the asset.

2. Multithreaded Backtesting:

• You can utilize the ThreadPoolExecutor and threading capabilities to download and backtest multiple assets simultaneously. This can significantly improve efficiency compared to a sequential approach.
• The custom_assets dictionary and its management functions will be crucial for providing asset symbols to the backtesting function within the thread pool.

Additional Notes:

• Remember to replace 'best_model_tcn_1sl_1tp_2p5SlTp_success.pkl' with the actual path to your trained model file.
• Consider error handling and logging mechanisms for potential issues during data download, backtesting calculations, or thread management.

#NOTE: Fetching from binance Futures perpetual USDT assets , if error 4xx accours, it means, there is some restriction from your government or VPN server is connected to restrcited area for binance to work. You can use assets from the collection given by me in next cell

import requests
def get_binance_futures_assets():
    url = "https://fapi.binance.com/fapi/v1/exchangeInfo"
    try:
        response = requests.get(url)
        response.raise_for_status()  # Raise an exception for 4xx and 5xx status codes
        data = response.json()
        assets = [asset['symbol'] for asset in data['symbols'] if asset['contractType'] == 'PERPETUAL' and asset['quoteAsset'] == 'USDT']
        return assets
    except requests.exceptions.RequestException as e:
        print("Failed to fetch Binance futures assets:", e)
        return []
# Get all Binance futures USDT perpetual assets
futures_assets = get_binance_futures_assets()
print("Binance Futures USDT Perpetual Assets:")
print(futures_assets, len(futures_assets))

output:
'BTCUSDT.P', 'ETHUSDT.P', 'BCHUSDT.P', 'XRPUSDT.P', 'EOSUSDT.P', 'LTCUSDT.P', 'TRXUSDT.P', 'ETCUSDT.P', 
        'LINKUSDT.P', 'XLMUSDT.P', 'ADAUSDT.P', 'XMRUSDT.P', 'DASHUSDT.P', 'ZECUSDT.P', 'XTZUSDT.P', 'BNBUSDT.P', 
        'ATOMUSDT.P', 'ONTUSDT.P', 'IOTAUSDT.P', 'BATUSDT.P', 'VETUSDT.P', 'NEOUSDT.P', 'QTUMUSDT.P', 'IOSTUSDT.P', 
        'THETAUSDT.P', 'ALGOUSDT.P', 'ZILUSDT.P', 'KNCUSDT.P', 'ZRXUSDT.P', 'COMPUSDT.P', 'OMGUSDT.P', 'DOGEUSDT.P', 
        'SXPUSDT.P', 'KAVAUSDT.P', 'BANDUSDT.P', 'RLCUSDT.P', 'WAVESUSDT.P', 'MKRUSDT.P', 'SNXUSDT.P', 'DOTUSDT.P', 
        'DEFIUSDT.P', 'YFIUSDT.P', 'BALUSDT.P', 'CRVUSDT.P', 'TRBUSDT.P', 'RUNEUSDT.P', 'SUSHIUSDT.P', 'SRMUSDT.P', 
        'EGLDUSDT.P', 'SOLUSDT.P', 'ICXUSDT.P', 'STORJUSDT.P', 'BLZUSDT.P', 'UNIUSDT.P', 'AVAXUSDT.P', 'FTMUSDT.P', 
        'HNTUSDT.P', 'ENJUSDT.P', 'FLMUSDT.P', 'TOMOUSDT.P', 'RENUSDT.P', 'KSMUSDT.P', 'NEARUSDT.P', 'AAVEUSDT.P', 
        'FILUSDT.P', 'RSRUSDT.P', 'LRCUSDT.P', 'MATICUSDT.P', 'OCEANUSDT.P', 'CVCUSDT.P', 'BELUSDT.P', 'CTKUSDT.P', 
        'AXSUSDT.P', 'ALPHAUSDT.P', 'ZENUSDT.P', 'SKLUSDT.P', 'GRTUSDT.P', '1INCHUSDT.P', 'CHZUSDT.P', 'SANDUSDT.P', 
        'ANKRUSDT.P', 'BTSUSDT.P', 'LITUSDT.P', 'UNFIUSDT.P', 'REEFUSDT.P', 'RVNUSDT.P', 'SFPUSDT.P', 'XEMUSDT.P', 
        'COTIUSDT.P', 'CHRUSDT.P', 'MANAUSDT.P', 'ALICEUSDT.P', 'HBARUSDT.P', 'ONEUSDT.P', 'LINAUSDT.P', 'STMXUSDT.P', 
        'DENTUSDT.P', 'CELRUSDT.P', 'HOTUSDT.P', 'MTLUSDT.P', 'OGNUSDT.P', 'NKNUSDT.P', 'SCUSDT.P', 'DGBUSDT.P', 
        '1000SHIBUSDT.P', 'BAKEUSDT.P', 'GTCUSDT.P', 'BTCDOMUSDT.P', 'IOTXUSDT.P', 'AUDIOUSDT.P', 'RAYUSDT.P', 'C98USDT.P', 
        'MASKUSDT.P', 'ATAUSDT.P', 'DYDXUSDT.P', '1000XECUSDT.P', 'GALAUSDT.P', 'CELOUSDT.P', 'ARUSDT.P', 'KLAYUSDT.P', 
        'ARPAUSDT.P', 'CTSIUSDT.P', 'LPTUSDT.P', 'ENSUSDT.P', 'PEOPLEUSDT.P', 'ANTUSDT.P', 'ROSEUSDT.P', 'DUSKUSDT.P', 
        'FLOWUSDT.P', 'IMXUSDT.P', 'API3USDT.P', 'GMTUSDT.P', 'APEUSDT.P', 'WOOUSDT.P', 'FTTUSDT.P', 'JASMYUSDT.P', 'DARUSDT.P', 
        'GALUSDT.P', 'OPUSDT.P', 'INJUSDT.P', 'STGUSDT.P', 'FOOTBALLUSDT.P', 'SPELLUSDT.P', '1000LUNCUSDT.P', 
        'LUNA2USDT.P', 'LDOUSDT.P', 'CVXUSDT.P', 'ICPUSDT.P', 'APTUSDT.P', 'QNTUSDT.P', 'BLUEBIRDUSDT.P', 'FETUSDT.P', 
        'FXSUSDT.P', 'HOOKUSDT.P', 'MAGICUSDT.P', 'TUSDT.P', 'RNDRUSDT.P', 'HIGHUSDT.P', 'MINAUSDT.P', 'ASTRUSDT.P', 
        'AGIXUSDT.P', 'PHBUSDT.P', 'GMXUSDT.P', 'CFXUSDT.P', 'STXUSDT.P', 'COCOSUSDT.P', 'BNXUSDT.P', 'ACHUSDT.P', 
        'SSVUSDT.P', 'CKBUSDT.P', 'PERPUSDT.P', 'TRUUSDT.P', 'LQTYUSDT.P', 'USDCUSDT.P', 'IDUSDT.P', 'ARBUSDT.P', 
        'JOEUSDT.P', 'TLMUSDT.P', 'AMBUSDT.P', 'LEVERUSDT.P', 'RDNTUSDT.P', 'HFTUSDT.P', 'XVSUSDT.P', 'BLURUSDT.P', 
        'EDUUSDT.P', 'IDEXUSDT.P', 'SUIUSDT.P', '1000PEPEUSDT.P', '1000FLOKIUSDT.P', 'UMAUSDT.P', 'RADUSDT.P', 
        'KEYUSDT.P', 'COMBOUSDT.P', 'NMRUSDT.P', 'MAVUSDT.P', 'MDTUSDT.P', 'XVGUSDT.P', 'WLDUSDT.P', 'PENDLEUSDT.P', 
        'ARKMUSDT.P', 'AGLDUSDT.P', 'YGGUSDT.P', 'DODOXUSDT.P', 'BNTUSDT.P', 'OXTUSDT.P', 'SEIUSDT.P', 'CYBERUSDT.P', 
        'HIFIUSDT.P', 'ARKUSDT.P', 'FRONTUSDT.P', 'GLMRUSDT.P', 'BICOUSDT.P', 'STRAXUSDT.P', 'LOOMUSDT.P', 'BIGTIMEUSDT.P', 
        'BONDUSDT.P', 'ORBSUSDT.P', 'STPTUSDT.P', 'WAXPUSDT.P', 'BSVUSDT.P', 'RIFUSDT.P', 'POLYXUSDT.P', 'GASUSDT.P', 
        'POWRUSDT.P', 'SLPUSDT.P', 'TIAUSDT.P', 'SNTUSDT.P', 'CAKEUSDT.P', 'MEMEUSDT.P', 'TWTUSDT.P', 'TOKENUSDT.P', 
        'ORDIUSDT.P', 'STEEMUSDT.P', 'BADGERUSDT.P', 'ILVUSDT.P', 'NTRNUSDT.P', 'MBLUSDT.P', 'KASUSDT.P', 'BEAMXUSDT.P', 
        '1000BONKUSDT.P', 'PYTHUSDT.P', 'SUPERUSDT.P', 'USTCUSDT.P', 'ONGUSDT.P', 'ETHWUSDT.P', 'JTOUSDT.P', '1000SATSUSDT.P', 
        'AUCTIONUSDT.P', '1000RATSUSDT.P', 'ACEUSDT.P', 'MOVRUSDT.P', 'NFPUSDT.P', 'AIUSDT.P', 'XAIUSDT.P', 
        'WIFUSDT.P', 'MANTAUSDT.P', 'ONDOUSDT.P', 'LSKUSDT.P', 'ALTUSDT.P', 'JUPUSDT.P', 'ZETAUSDT.P', 'RONINUSDT.P', 
        'DYMUSDT.P', 'OMUSDT.P', 'PIXELUSDT.P', 'STRKUSDT.P', 'MAVIAUSDT.P', 'GLMUSDT.P', 'PORTALUSDT.P', 'TONUSDT.P', 
        'AXLUSDT.P', 'MYROUSDT.P', 'METISUSDT.P', 'AEVOUSDT.P', 'VANRYUSDT.P', 'BOMEUSDT.P', 'ETHFIUSDT.P', 'ENAUSDT.P', 
        'WUSDT.P', 'TNSRUSDT.P', 'SAGAUSDT.P', 'TAOUSDT.P', 'OMNIUSDT.P', 'REZUSDT.P'

This code snippet retrieves a list of perpetual USDT contracts available on Binance Futures using the official Binance API. Here’s a breakdown:

Function:

get_binance_futures_assets function:

• Defines the API endpoint URL for retrieving exchange information.
• Uses a try-except block to handle potential errors during the request.

Within the try block:

• Makes a GET request to the Binance API endpoint.
• Raises an exception for status codes in the 4xx (client errors) or 5xx (server errors) range to indicate failures.
• Parses the JSON response from the successful request.

Extracts symbols from the response data:

• Iterates through the 'symbols' list in the JSON data.

Filters for assets with these criteria:

• 'contractType' is 'PERPETUAL' (indicates perpetual contracts).
• 'quoteAsset' is 'USDT' (indicates USDT-quoted contracts).
• Creates a list of asset symbols meeting the criteria and returns it.
• The except block catches potential request exceptions and prints an error message. It also returns an empty list in case of failures.

Printing Results:

• Calls the get_binance_futures_assets function to retrieve the asset list.
• Prints a message indicating the retrieved assets and their count.

Additional Notes:

• This approach leverages the official Binance API, which might be subject to rate limits or changes in the future. Consider implementing appropriate error handling and retry mechanisms.
• The code assumes a successful API call. You might want to add checks for specific error codes (e.g., 429 for “Too Many Requests”) and handle them gracefully (e.g., retrying after a delay).

# !pip install --upgrade --no-cache-dir git+https://github.com/rongardF/tvdatafeed.git

import os
import json
import asyncio
from datetime import datetime, timedelta
import pandas as pd
from tvDatafeed import TvDatafeed, Interval
# Initialize TvDatafeed object
# username = 'YourTradingViewUsername'
# password = 'YourTradingViewPassword'
# tv = TvDatafeed(username, password)
tv = TvDatafeed()
timeframe = '15m'
interval = None
if timeframe == '1m':
    interval = Interval.in_1_minute
elif timeframe == '3m':
    interval = Interval.in_3_minute
elif timeframe == '5m':
    interval = Interval.in_5_minute
elif timeframe == '15m':
    interval = Interval.in_15_minute
elif timeframe == '30m':
    interval = Interval.in_30_minute
elif timeframe == '45m':
    interval = Interval.in_45_minute
elif timeframe == '1h':
    interval = Interval.in_1_hour
elif timeframe == '2h':
    interval = Interval.in_2_hour
elif timeframe == '4h':
    interval = Interval.in_4_hour
elif timeframe == '1d':
    interval = Interval.in_daily
elif timeframe == '1w':
    interval = Interval.in_weekly
elif timeframe == '1M':
    interval = Interval.in_monthly
# NOTE: List of symbols around 126 mentioned here. You can change to your own set of lists if you know the tradingview code for the symbol you want to download.
data = [
    'BTCUSDT.P', 'ETHUSDT.P', 'BCHUSDT.P', 'XRPUSDT.P', 'EOSUSDT.P', 'LTCUSDT.P', 'TRXUSDT.P', 'ETCUSDT.P', 
        'LINKUSDT.P', 'XLMUSDT.P', 'ADAUSDT.P', 'XMRUSDT.P', 'DASHUSDT.P', 'ZECUSDT.P', 'XTZUSDT.P', 'BNBUSDT.P', 
        'ATOMUSDT.P', 'ONTUSDT.P', 'IOTAUSDT.P', 'BATUSDT.P', 'VETUSDT.P', 'NEOUSDT.P', 'QTUMUSDT.P', 'IOSTUSDT.P', 
        'THETAUSDT.P', 'ALGOUSDT.P', 'ZILUSDT.P', 'KNCUSDT.P', 'ZRXUSDT.P', 'COMPUSDT.P', 'OMGUSDT.P', 'DOGEUSDT.P', 
        'SXPUSDT.P', 'KAVAUSDT.P', 'BANDUSDT.P', 'RLCUSDT.P', 'WAVESUSDT.P', 'MKRUSDT.P', 'SNXUSDT.P', 'DOTUSDT.P', 
        'DEFIUSDT.P', 'YFIUSDT.P', 'BALUSDT.P', 'CRVUSDT.P', 'TRBUSDT.P', 'RUNEUSDT.P', 'SUSHIUSDT.P', 'SRMUSDT.P', 
        'EGLDUSDT.P', 'SOLUSDT.P', 'ICXUSDT.P', 'STORJUSDT.P', 'BLZUSDT.P', 'UNIUSDT.P', 'AVAXUSDT.P', 'FTMUSDT.P', 
        'HNTUSDT.P', 'ENJUSDT.P', 'FLMUSDT.P', 'TOMOUSDT.P', 'RENUSDT.P', 'KSMUSDT.P', 'NEARUSDT.P', 'AAVEUSDT.P', 
        'FILUSDT.P', 'RSRUSDT.P', 'LRCUSDT.P', 'MATICUSDT.P', 'OCEANUSDT.P', 'CVCUSDT.P', 'BELUSDT.P', 'CTKUSDT.P', 
        'AXSUSDT.P', 'ALPHAUSDT.P', 'ZENUSDT.P', 'SKLUSDT.P', 'GRTUSDT.P', '1INCHUSDT.P', 'CHZUSDT.P', 'SANDUSDT.P', 
        'ANKRUSDT.P', 'BTSUSDT.P', 'LITUSDT.P', 'UNFIUSDT.P', 'REEFUSDT.P', 'RVNUSDT.P', 'SFPUSDT.P', 'XEMUSDT.P', 
        'COTIUSDT.P', 'CHRUSDT.P', 'MANAUSDT.P', 'ALICEUSDT.P', 'HBARUSDT.P', 'ONEUSDT.P', 'LINAUSDT.P', 'STMXUSDT.P', 
        'DENTUSDT.P', 'CELRUSDT.P', 'HOTUSDT.P', 'MTLUSDT.P', 'OGNUSDT.P', 'NKNUSDT.P', 'SCUSDT.P', 'DGBUSDT.P', 
        '1000SHIBUSDT.P', 'BAKEUSDT.P', 'GTCUSDT.P', 'BTCDOMUSDT.P', 'IOTXUSDT.P', 'AUDIOUSDT.P', 'RAYUSDT.P', 'C98USDT.P', 
        'MASKUSDT.P', 'ATAUSDT.P', 'DYDXUSDT.P', '1000XECUSDT.P', 'GALAUSDT.P', 'CELOUSDT.P', 'ARUSDT.P', 'KLAYUSDT.P', 
        'ARPAUSDT.P', 'CTSIUSDT.P', 'LPTUSDT.P', 'ENSUSDT.P', 'PEOPLEUSDT.P', 'ANTUSDT.P', 'ROSEUSDT.P', 'DUSKUSDT.P', 
        'FLOWUSDT.P', 'IMXUSDT.P', 'API3USDT.P', 'GMTUSDT.P', 'APEUSDT.P', 'WOOUSDT.P', 'FTTUSDT.P', 'JASMYUSDT.P', 'DARUSDT.P', 
        'GALUSDT.P', 'OPUSDT.P', 'INJUSDT.P', 'STGUSDT.P', 'FOOTBALLUSDT.P', 'SPELLUSDT.P', '1000LUNCUSDT.P', 
        'LUNA2USDT.P', 'LDOUSDT.P', 'CVXUSDT.P', 'ICPUSDT.P', 'APTUSDT.P', 'QNTUSDT.P', 'BLUEBIRDUSDT.P', 'FETUSDT.P', 
        'FXSUSDT.P', 'HOOKUSDT.P', 'MAGICUSDT.P', 'TUSDT.P', 'RNDRUSDT.P', 'HIGHUSDT.P', 'MINAUSDT.P', 'ASTRUSDT.P', 
        'AGIXUSDT.P', 'PHBUSDT.P', 'GMXUSDT.P', 'CFXUSDT.P', 'STXUSDT.P', 'COCOSUSDT.P', 'BNXUSDT.P', 'ACHUSDT.P', 
        'SSVUSDT.P', 'CKBUSDT.P', 'PERPUSDT.P', 'TRUUSDT.P', 'LQTYUSDT.P', 'USDCUSDT.P', 'IDUSDT.P', 'ARBUSDT.P', 
        'JOEUSDT.P', 'TLMUSDT.P', 'AMBUSDT.P', 'LEVERUSDT.P', 'RDNTUSDT.P', 'HFTUSDT.P', 'XVSUSDT.P', 'BLURUSDT.P', 
        'EDUUSDT.P', 'IDEXUSDT.P', 'SUIUSDT.P', '1000PEPEUSDT.P', '1000FLOKIUSDT.P', 'UMAUSDT.P', 'RADUSDT.P', 
        'KEYUSDT.P', 'COMBOUSDT.P', 'NMRUSDT.P', 'MAVUSDT.P', 'MDTUSDT.P', 'XVGUSDT.P', 'WLDUSDT.P', 'PENDLEUSDT.P', 
        'ARKMUSDT.P', 'AGLDUSDT.P', 'YGGUSDT.P', 'DODOXUSDT.P', 'BNTUSDT.P', 'OXTUSDT.P', 'SEIUSDT.P', 'CYBERUSDT.P', 
        'HIFIUSDT.P', 'ARKUSDT.P', 'FRONTUSDT.P', 'GLMRUSDT.P', 'BICOUSDT.P', 'STRAXUSDT.P', 'LOOMUSDT.P', 'BIGTIMEUSDT.P', 
        'BONDUSDT.P', 'ORBSUSDT.P', 'STPTUSDT.P', 'WAXPUSDT.P', 'BSVUSDT.P', 'RIFUSDT.P', 'POLYXUSDT.P', 'GASUSDT.P', 
        'POWRUSDT.P', 'SLPUSDT.P', 'TIAUSDT.P', 'SNTUSDT.P', 'CAKEUSDT.P', 'MEMEUSDT.P', 'TWTUSDT.P', 'TOKENUSDT.P', 
        'ORDIUSDT.P', 'STEEMUSDT.P', 'BADGERUSDT.P', 'ILVUSDT.P', 'NTRNUSDT.P', 'MBLUSDT.P', 'KASUSDT.P', 'BEAMXUSDT.P', 
        '1000BONKUSDT.P', 'PYTHUSDT.P', 'SUPERUSDT.P', 'USTCUSDT.P', 'ONGUSDT.P', 'ETHWUSDT.P', 'JTOUSDT.P', '1000SATSUSDT.P', 
        'AUCTIONUSDT.P', '1000RATSUSDT.P', 'ACEUSDT.P', 'MOVRUSDT.P', 'NFPUSDT.P', 'AIUSDT.P', 'XAIUSDT.P', 
        'WIFUSDT.P', 'MANTAUSDT.P', 'ONDOUSDT.P', 'LSKUSDT.P', 'ALTUSDT.P', 'JUPUSDT.P', 'ZETAUSDT.P', 'RONINUSDT.P', 
        'DYMUSDT.P', 'OMUSDT.P', 'PIXELUSDT.P', 'STRKUSDT.P', 'MAVIAUSDT.P', 'GLMUSDT.P', 'PORTALUSDT.P', 'TONUSDT.P', 
        'AXLUSDT.P', 'MYROUSDT.P', 'METISUSDT.P', 'AEVOUSDT.P', 'VANRYUSDT.P', 'BOMEUSDT.P', 'ETHFIUSDT.P', 'ENAUSDT.P', 
        'WUSDT.P', 'TNSRUSDT.P', 'SAGAUSDT.P', 'TAOUSDT.P', 'OMNIUSDT.P', 'REZUSDT.P'
]
nest_asyncio.apply()
# Define data download function
async def download_data(symbol):
    try:
        data = tv.get_hist(symbol=symbol, exchange='BINANCE', interval=interval, n_bars=20000, extended_session=True)
        if not data.empty:
            # Convert Date objects to strings
            # data['Date'] = data.index.date.astype(str)
            # data['Time'] = data.index.time.astype(str)
            data['date'] = data.index.astype(str)  # Add a new column for timestamps
            folder_name = f"tradingview_crypto_assets_{timeframe}"
            os.makedirs(folder_name, exist_ok=True)
            # Replace "USDT.P" with "/USDT:USDT" in the file name
            symbol_file_name = symbol.replace("USDT.P", "") + ".json"
            file_name = os.path.join(folder_name, symbol_file_name)
            # Convert DataFrame to dictionary
            data_dict = data.to_dict(orient='records')
            with open(file_name, "w") as file:
                # Serialize dictionary to JSON
                json.dump(data_dict, file)
            print(f"Data for {symbol} downloaded and saved successfully.")
        else:
            print(f"No data available for {symbol}.")
    except Exception as e:
        print(f"Error occurred while downloading data for {symbol}: {e}")
# Define main function to run async download tasks
async def main():
    tasks = [download_data(symbol) for symbol in data]
    await asyncio.gather(*tasks)
# Run the main function
asyncio.run(main())

This code snippet demonstrates how to download historical cryptocurrency data from TradingView for multiple assets using the tvDatafeed library. Here's a breakdown:

Imports:

• Includes libraries for asynchronous programming (asyncio), working with dates (datetime), data manipulation (pandas), and file handling (os, json).
• Imports the TvDatafeed class from tvDatafeed for interacting with TradingView.

TvDatafeed Object:

• Initializes a TvDatafeed object (tv) without username and password (assuming a free account). Paid accounts might require credentials.

Timeframe and Interval:

• Sets the desired timeframe (timeframe) for data download (e.g., "15m" for 15-minute intervals).
• Maps the timeframe to the corresponding Interval enumeration value using a series of if statements.

Symbols List:

• Defines a long list of symbols (data) representing cryptocurrencies on Binance Futures with perpetual USDT contracts (identified by ".P" suffix).

Asynchronous Programming Setup:

• Initializes nest_asyncio.apply() to enable the use of asynchronous functions within a non-asynchronous context.

Download Function:

• Defines an asynchronous function download_data(symbol) that takes a symbol as input.

Attempts to download historical data for the symbol using tv.get_hist:

• Specifies the symbol, exchange (“BINANCE”), interval, number of bars (20000), and extended session (to potentially capture pre-market/after-market data).
• Checks if downloaded data (data) is not empty.

If data is available:

• Converts the index (timestamps) to strings in a new column named “date”.
• Creates a folder named tradingview_crypto_assets_{timeframe} to store the downloaded data (creates it if it doesn't exist).
• Constructs the filename by replacing “.P” with “/USDT:USDT” in the symbol and appending “.json”.
• Converts the DataFrame to a dictionary using to_dict(orient='records').
• Saves the dictionary as JSON to the constructed filename.
• Prints a success message.

If no data is available:

• Prints a message indicating no data for the symbol.
• Catches any exceptions (Exception) during download and prints an error message with the exception details.

Main Function:

• Defines an asynchronous function main that:
• Creates a list of asynchronous tasks (tasks) using list comprehension. Each task calls download_data for a symbol from the data list.
• Uses asyncio.gather(*tasks) to run all download tasks concurrently.

Running the Download:

• Uses asyncio.run(main()) to execute the asynchronous tasks within the main function.

Important Notes:

• This code retrieves data for a large number of symbols. Downloading a significant amount of data might exceed free account limitations or take a long time. Consider rate limits and adjust accordingly.
• The code assumes a specific symbol format with the “.P” suffix. You might need to modify it for different symbol formats.
• Error handling can be improved by implementing specific checks for different exception types (e.g., network errors, API errors).

Hyperoptimization of Multiple Assets for Specific ML/DL Model:

from pandas import Timestamp

# Define a function to process each JSON file
def process_json(file_path):
    # try:
    with open(file_path, "r") as f:
        data = json.load(f)
    df = pd.DataFrame(data)
    df.rename(columns={'date': "Date", 'open': "Open", 'high': "High", 'low': "Low", 'close': "Adj Close", 'volume': "Volume"}, inplace=True)
    df["Date"] = pd.to_datetime(df['Date'])
    df.set_index("Date", inplace=True)
    df['Close'] = df['Adj Close']
    symbol_name = df['symbol'].iloc[0]  # Assuming all rows have the same symbol
    symbol_name = symbol_name.replace("BINANCE:", "")
    symbol_name = symbol_name.replace("USDT.P", "/USDT:USDT")
    df.drop(columns=['symbol'], inplace=True)
    target_prediction_number = 2
    time_periods = [6, 8, 10, 12, 14, 16, 18, 22, 26, 33, 44, 55]
    name_periods = [6, 8, 10, 12, 14, 16, 18, 22, 26, 33, 44, 55]
    new_columns = []
    for period in time_periods:
        for nperiod in name_periods:
            df[f'ATR_{period}'] = ta.ATR(df['High'], df['Low'], df['Close'], timeperiod=period)
            df[f'EMA_{period}'] = ta.EMA(df['Close'], timeperiod=period)
            df[f'RSI_{period}'] = ta.RSI(df['Close'], timeperiod=period)
            df[f'VWAP_{period}'] = ta.SUM(df['Volume'] * (df['High'] + df['Low'] + df['Close']) / 3, timeperiod=period) / ta.SUM(df['Volume'], timeperiod=period)
            df[f'ROC_{period}'] = ta.ROC(df['Close'], timeperiod=period)
            df[f'KC_upper_{period}'] = ta.EMA(df['High'], timeperiod=period)
            df[f'KC_middle_{period}'] = ta.EMA(df['Low'], timeperiod=period)
            df[f'Donchian_upper_{period}'] = ta.MAX(df['High'], timeperiod=period)
            df[f'Donchian_lower_{period}'] = ta.MIN(df['Low'], timeperiod=period)
            macd, macd_signal, _ = ta.MACD(df['Close'], fastperiod=(period + 12), slowperiod=(period + 26), signalperiod=(period + 9))
            df[f'MACD_{period}'] = macd
            df[f'MACD_signal_{period}'] = macd_signal
            bb_upper, bb_middle, bb_lower = ta.BBANDS(df['Close'], timeperiod=period, nbdevup=2, nbdevdn=2)
            df[f'BB_upper_{period}'] = bb_upper
            df[f'BB_middle_{period}'] = bb_middle
            df[f'BB_lower_{period}'] = bb_lower
            df[f'EWO_{period}'] = ta.SMA(df['Close'], timeperiod=(period+5)) - ta.SMA(df['Close'], timeperiod=(period+35))
    df["Returns"] = (df["Adj Close"] / df["Adj Close"].shift(target_prediction_number)) - 1
    df["Range"] = (df["High"] / df["Low"]) - 1
    df["Volatility"] = df['Returns'].rolling(window=target_prediction_number).std()
    # Volume-Based Indicators
    df['OBV'] = ta.OBV(df['Close'], df['Volume'])
    df['ADL'] = ta.AD(df['High'], df['Low'], df['Close'], df['Volume'])

    # Momentum-Based Indicators
    df['Stoch_Oscillator'] = ta.STOCH(df['High'], df['Low'], df['Close'])[0]
    df['PSAR'] = ta.SAR(df['High'], df['Low'], acceleration=0.02, maximum=0.2)
    # More feature engineering...
    timeframe_diff = df.index[-1] - df.index[-2]
    timeframe = None
    # Define timeframe based on time difference
    if timeframe_diff == pd.Timedelta(minutes=1):
        timeframe = '1m'
    elif timeframe_diff == pd.Timedelta(minutes=3):
        timeframe = '3m'
    elif timeframe_diff == pd.Timedelta(minutes=5):
        timeframe = '5m'
    elif timeframe_diff == pd.Timedelta(minutes=15):
        timeframe = '15m'
    elif timeframe_diff == pd.Timedelta(minutes=30):
        timeframe = '30m'
    elif timeframe_diff == pd.Timedelta(minutes=45):
        timeframe = '45m'
    elif timeframe_diff == pd.Timedelta(hours=1):
        timeframe = '1h'
    elif timeframe_diff == pd.Timedelta(days=1):
        timeframe = '1d'
    elif timeframe_diff == pd.Timedelta(weeks=1):
        timeframe = '1w'
    else:
        timeframe = 'Not sure'
        
    # print('timeframe is - ', timeframe)
    # Remove rows containing inf or nan values
    df.dropna(inplace=True)
    # Scaling
    scaler = MinMaxScaler(feature_range=(0,1))
    X = df.copy()
    X_scale = scaler.fit_transform(X)

    # Define a function to reshape the data
    def reshape_data(data, time_steps):
        samples = len(data) - time_steps + 1
        reshaped_data = np.zeros((samples, time_steps, data.shape[1]))
        for i in range(samples):
            reshaped_data[i] = data[i:i + time_steps]
        return reshaped_data
    # Reshape the scaled X data
    time_steps = 1  # Adjust the number of time steps as needed
    X_reshaped = reshape_data(X_scale, time_steps)
    # Now X_reshaped has the desired three-dimensional shape: (samples, time_steps, features)
    # Each sample contains scaled data for a specific time window
    X = X_reshaped
    # Use the loaded model to predict on the entire dataset
    df_ens = df.copy() 
    # df_ens['voting_classifier_ensembel_with_scale'] = np.argmax(model.predict(X), axis=1)
    df_ens['voting_classifier_ensembel_with_scale'] = np.argmax(model.predict(X), axis=2)
    df_ens['vcews'] = df_ens['voting_classifier_ensembel_with_scale'].shift(0).dropna().astype(int)
    df_ens = df_ens.dropna()
    # Backtesting
    df_ens = df_ens.reset_index(inplace=False)
    df_ens['Date'] = pd.to_datetime(df_ens['Date'])
    df_ens.set_index('Date', inplace=True)
    best_params = {'Optimizer': 'Return [%]',
        'model_trained_on': model_name,
        'OptimizerResult_Cross': 617.5341106880867,
        'BEST_STOP_LOSS_sl_pct_long': 15,
        'BEST_TAKE_PROFIT_tp_pct_long': 25,
        'BEST_LIMIT_ORDER_limit_long': 24,
        'BEST_STOP_LOSS_sl_pct_short': 15,
        'BEST_TAKE_PROFIT_tp_pct_short': 25,
        'BEST_LIMIT_ORDER_limit_short': 24,
        'BEST_LEVERAGE_margin_leverage': 1,
        'TRAILING_ACTIVATE_PCT': 10,
        'TRAILING_STOP_PCT' : 5,
        'roi_at_50' : 24,
        'roi_at_100' : 20,
        'roi_at_150' : 18,
        'roi_at_200' : 15,
        'roi_at_300' : 13,
        'roi_at_500' : 10}
    # Define SIGNAL_3 function
    def SIGNAL_3(df_ens):
        return df_ens['vcews']
    # Define MyCandlesStrat_3 class
    class MyCandlesStrat_3(Strategy):  
        sl_pct_l = best_params['BEST_STOP_LOSS_sl_pct_long'] 
        tp_pct_l = best_params['BEST_TAKE_PROFIT_tp_pct_long'] 
        limit_l = best_params['BEST_LIMIT_ORDER_limit_long'] 
        sl_pct_s = best_params['BEST_STOP_LOSS_sl_pct_short'] 
        tp_pct_s = best_params['BEST_TAKE_PROFIT_tp_pct_short'] 
        limit_s = best_params['BEST_LIMIT_ORDER_limit_short'] 
        margin_leverage = best_params['BEST_LEVERAGE_margin_leverage']
        TRAILING_ACTIVATE_PCT = best_params['TRAILING_ACTIVATE_PCT']
        TRAILING_STOP_PCT = best_params['TRAILING_STOP_PCT']
        roi_at_50 = best_params['roi_at_50']
        roi_at_100 = best_params['roi_at_100']
        roi_at_150 = best_params['roi_at_150']
        roi_at_200 = best_params['roi_at_200']
        roi_at_300 = best_params['roi_at_300']
        roi_at_500 = best_params['roi_at_500']
        def init(self):
            super().init()
            self.signal1 = self.I(SIGNAL_3, self.data)
            self.entry_time = Timestamp.now()
            self.max_profit = 0
        def next(self):
            super().next() 
            if (self.signal1 == 1):
                
                sl_price = self.data.Close[-1] * (1 - (self.sl_pct_l * 0.001))
                tp_price = self.data.Close[-1] * (1 + (self.tp_pct_l * 0.001))
                limit_price_l = tp_price * 0.994
                self.position.is_long
                self.buy(sl=sl_price, limit=limit_price_l, tp=tp_price)
                
                if self.position.is_long:
                    self.entry_time = self.trades[0].entry_time  # Accessing the current datetime
                
                # Calculate current profit
                # current_profit = self.trades[0].pl_pct
                # Check for trailing stop loss based on current profit
                if self.position and self.trades[0].pl_pct >= (self.TRAILING_ACTIVATE_PCT * 0.001):
                    self.max_profit = max(self.max_profit, self.trades[0].pl_pct)
                    trailing_stop_price = self.trades[0].entry_price * (1 + (self.max_profit - (self.TRAILING_STOP_PCT * 0.001)))
                    sl_price = min((self.data.Close[-1] * (1 - (self.TRAILING_STOP_PCT * 0.001))), trailing_stop_price)
                    time_spent_by_asset1 = (self.data.index[-1] - self.trades[0].entry_time).total_seconds() / 60
                    # Check for time interval-based selling
                    if self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 50) and (self.trades[0].pl_pct > (self.roi_at_50 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 50) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 100) and (self.trades[0].pl_pct > (self.roi_at_100 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 100) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 150) and (self.trades[0].pl_pct > (self.roi_at_150 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 150) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 200) and (self.trades[0].pl_pct > (self.roi_at_200 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 200) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 300) and (self.trades[0].pl_pct > (self.roi_at_300 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 300) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166< 950) and (self.trades[0].pl_pct > (self.roi_at_500 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166>= 950):
                        self.position.close()
            elif (self.signal1 == 2):
                
                sl_price = self.data.Close[-1] * (1 + (self.sl_pct_s * 0.001))
                tp_price = self.data.Close[-1] * (1 - (self.tp_pct_s * 0.001))
                limit_price_s = tp_price * 1.004
                self.position.is_short
                self.sell(sl=sl_price, limit=limit_price_s, tp=tp_price)
                
                if self.position.is_short:
                    self.entry_time = self.trades[0].entry_time  # Accessing the current datetime
                
                # Calculate current profit
                # current_profit = self.trades[0].pl_pct
                # Check for trailing stop loss based on current profit
                if self.position and self.trades[0].pl_pct >= (self.TRAILING_ACTIVATE_PCT * 0.001):
                    self.max_profit = max(self.max_profit, self.trades[0].pl_pct)
                    trailing_stop_price = self.trades[0].entry_price * (1 - (self.max_profit - (self.TRAILING_STOP_PCT * 0.001)))
                    sl_price = max((self.data.Close[-1] * (1 - (self.TRAILING_STOP_PCT * 0.001))), trailing_stop_price)
                    time_spent_by_asset1 = (self.data.index[-1] - self.trades[0].entry_time).total_seconds() / 60
                # Check for time interval-based selling
                if self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 50) and (self.trades[0].pl_pct > (self.roi_at_50 * 0.001)):
                    self.position.close()
                elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 50) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 100) and (self.trades[0].pl_pct > (self.roi_at_100 * 0.001)):
                    self.position.close()
                elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 100) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 150) and (self.trades[0].pl_pct > (self.roi_at_150 * 0.001)):
                    self.position.close()
                elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 150) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 200) and (self.trades[0].pl_pct > (self.roi_at_200 * 0.001)):
                    self.position.close()
                elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 200) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 300) and (self.trades[0].pl_pct > (self.roi_at_300 * 0.001)):
                    self.position.close()
                elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 300) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166< 950) and (self.trades[0].pl_pct > (self.roi_at_500 * 0.001)):
                    self.position.close()
                elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166>= 950):
                    self.position.close()

    # Run backtest
    bt_3 = Backtest(df_ens, MyCandlesStrat_3, cash=100000, commission=.001, margin= (1/MyCandlesStrat_3.margin_leverage), exclusive_orders=False)
    stat_3 = bt_3.run()
    print("backtest one done at 226 line - ", stat_3)
    # custom_assets = {}
    if ((stat_3['Return [%]'] > (stat_3['Buy & Hold Return [%]'] * 3)) 
        & (stat_3['Profit Factor'] > 1.0) 
        & (stat_3['Max. Drawdown [%]'] > -40)
        & (stat_3['Win Rate [%]'] > 55)
        & (stat_3['Return [%]'] > 0)):
        file_prefix = file_path.split('/')[-1].split('.')[0]
        
        best_params = {'Optimizer': '1st backtest - Expectancy',
                       'model_trained_on': model_name,
        'OptimizerResult_Cross': f"For {file_prefix}/USDT:USDT backtest was done from {stat_3['Start']} upto {stat_3['End']} for a duration of {stat_3['Duration']} using time frame of {timeframe} with Win Rate % - {round(stat_3['Win Rate [%]'],2)}, Return % - {round(stat_3['Return [%]'],3)},Expectancy % - {round(stat_3['Expectancy [%]'],5)} and Sharpe Ratio - {round(stat_3['Sharpe Ratio'],4)}.",
        'BEST_STOP_LOSS_sl_pct_long': 15,
        'BEST_TAKE_PROFIT_tp_pct_long': 25,
        'BEST_LIMIT_ORDER_limit_long': 24,
        'BEST_STOP_LOSS_sl_pct_short': 15,
        'BEST_TAKE_PROFIT_tp_pct_short': 25,
        'BEST_LIMIT_ORDER_limit_short': 24,
        'BEST_LEVERAGE_margin_leverage': 1,
        'TRAILING_ACTIVATE_PCT': 10,
        'TRAILING_STOP_PCT' : 5,
        'roi_at_50' : 24,
        'roi_at_100' : 20,
        'roi_at_150' : 18,
        'roi_at_200' : 15,
        'roi_at_300' : 13,
        'roi_at_500' : 10}
        key_mapping = {
            'Optimizer': 'Optimizer_used',
            'model_trained_on': 'model_name',
            'OptimizerResult_Cross': 'Optimizer_result',
            'BEST_STOP_LOSS_sl_pct_long': 'stop_loss_percent_long',
            'BEST_TAKE_PROFIT_tp_pct_long': 'take_profit_percent_long',
            'BEST_LIMIT_ORDER_limit_long': 'limit_long',
            'BEST_STOP_LOSS_sl_pct_short': 'stop_loss_percent_short',
            'BEST_TAKE_PROFIT_tp_pct_short': 'take_profit_percent_short',
            'BEST_LIMIT_ORDER_limit_short': 'limit_short',
            'BEST_LEVERAGE_margin_leverage': 'margin_leverage',
            'TRAILING_ACTIVATE_PCT': 'TRAILING_ACTIVATE_PCT',
            'TRAILING_STOP_PCT' : 'TRAILING_STOP_PCT',
            'roi_at_50' : 'roi_at_50',
            'roi_at_100' : 'roi_at_100',
            'roi_at_150' :'roi_at_150',
            'roi_at_200' : 'roi_at_200',
            'roi_at_300' : 'roi_at_300',
            'roi_at_500' : 'roi_at_500'
        }
        custom_assets = load_custom_assets()
        transformed_params = {}
        for old_key, value in best_params.items():
            new_key = key_mapping.get(old_key, old_key)
            transformed_params[new_key] = value
        new_key = file_prefix + "/USDT:USDT"
        # custom_assets[new_key] = transformed_params
        # Update or add new entry to custom_assets
        if new_key in custom_assets:
            # Update existing entry
            for key, value in transformed_params.items():
                if isinstance(value, (int, float)) and key != 'margin_leverage' and value >= 1:
                    transformed_params[key] = round(transformed_params[key] * 0.001, 5)
            custom_assets[new_key].update(transformed_params)
        else:
            # Add new entry
            # Multiply numerical values by 0.001 for new entry if value > 1
            for key, value in transformed_params.items():
                if isinstance(value, (int, float)) and key != 'margin_leverage' and value >= 1:
                    transformed_params[key] = round(transformed_params[key] * 0.001, 5)
            custom_assets[new_key] = transformed_params
        
        # Save custom_assets to JSON file
        save_custom_assets(custom_assets)
        print(custom_assets)
    else:
        # Optimization
        def optimize_strategy():
            # Optimization Params
            optimizer = 'Win Rate [%]'
            stats = bt_3.optimize(
                sl_pct_l = range(6,100, 2), # (5,10,15,20,25,30,40,50,75,100)
                tp_pct_l =  range(40,100, 2), # (0.005, 0.01, 0.015, 0.02, 0.025, 0.03, 0.04, 0.05, 0.075, 0.1)
                # limit_l =  (4,9,14,19,24,29,39,49,74,90),#  (0.004, 0.009, 0.014, 0.019, 0.024, 0.029, 0.039, 0.049, 0.074, 0.09)
                sl_pct_s = range(6,100, 2),
                tp_pct_s =  range(40,100, 2),
                # limit_s =  (4,9,14,19,24,29,39,49,74,90),
                margin_leverage = range(1, 8),
                TRAILING_ACTIVATE_PCT = range(6,100,2),
                TRAILING_STOP_PCT = range(6,100,2),
                roi_at_50 = range(6,100,2),
                roi_at_100 = range(6,100,2),
                roi_at_150 = range(6,100,2),
                roi_at_200 = range(6,100,2),
                roi_at_300 = range(6,100,2),
                roi_at_500 = range(6,100,2),
                constraint=lambda p: ( (p.sl_pct_l > (p.tp_pct_l) ) and 
                                      ((p.sl_pct_s) > (p.tp_pct_s)) and 
                                      (p.roi_at_50 > p.roi_at_100) and (p.roi_at_100 > p.roi_at_150) and 
                                      (p.roi_at_150 > p.roi_at_200) and (p.roi_at_200 > p.roi_at_300) and (p.roi_at_300 > p.roi_at_500) and
                                     (p.TRAILING_ACTIVATE_PCT > p.TRAILING_STOP_PCT)),
                maximize = optimizer,
                return_optimization=True,
                method = 'skopt',
                max_tries = 120 # 20% for 0.2 and 100% for 1.0, this applys when not using 'skopt' method, for 'skopt' number starts from 1 to 200 max epochs 
            )
            # Extract the optimization results
            best_params = {
                'Optimizer': optimizer,
                'model_trained_on': model_name,
                'OptimizerResult_Cross': stats[0][optimizer],
                'BEST_STOP_LOSS_sl_pct_long': stats[1].x[0],
                'BEST_TAKE_PROFIT_tp_pct_long': stats[1].x[1] ,
                'BEST_LIMIT_ORDER_limit_long': stats[1].x[1] * 0.997,
                'BEST_STOP_LOSS_sl_pct_short': stats[1].x[2] ,
                'BEST_TAKE_PROFIT_tp_pct_short': stats[1].x[3] ,
                'BEST_LIMIT_ORDER_limit_short': stats[1].x[3] * 0.997,
                'BEST_LEVERAGE_margin_leverage': stats[1].x[4],
                'TRAILING_ACTIVATE_PCT': stats[1].x[5],
                'TRAILING_STOP_PCT' : stats[1].x[6],
                'roi_at_50' : stats[1].x[7],
                'roi_at_100' : stats[1].x[8],
                'roi_at_150' : stats[1].x[9],
                'roi_at_200' : stats[1].x[10],
                'roi_at_300' : stats[1].x[11],
                'roi_at_500' : stats[1].x[12]
                # 'BEST_STOP_LOSS_sl_pct_long': stats._strategy.sl_pct_l,
                # 'BEST_TAKE_PROFIT_tp_pct_long': stats._strategy.tp_pct_l,
                # 'BEST_LIMIT_ORDER_limit_long': stats._strategy.tp_pct_l * 0.998,
                # 'BEST_STOP_LOSS_sl_pct_short': stats._strategy.sl_pct_s,
                # 'BEST_TAKE_PROFIT_tp_pct_short': stats._strategy.tp_pct_s,
                # 'BEST_LIMIT_ORDER_limit_short': stats._strategy.sl_pct_s * 0.998,
                # 'BEST_LEVERAGE_margin_leverage': stats._strategy.margin_leverage
            }
            
            return best_params

        # Obtain best parameters
        best_params = optimize_strategy()
        print("best_params line 322 ", best_params)
        if best_params:
            print(best_params)
        else:
            best_params = {'Optimizer': 'Return [%]',
                           'model_trained_on': model_name,
            'OptimizerResult_Cross': 617.5341106880867,
            'BEST_STOP_LOSS_sl_pct_long': 0.025,
            'BEST_TAKE_PROFIT_tp_pct_long': 0.025,
            'BEST_LIMIT_ORDER_limit_long': 0.024,
            'BEST_STOP_LOSS_sl_pct_short': 0.025,
            'BEST_TAKE_PROFIT_tp_pct_short': 0.025,
            'BEST_LIMIT_ORDER_limit_short': 0.024,
            'TRAILING_ACTIVATE_PCT': 10,
            'TRAILING_STOP_PCT' : 5,
            'roi_at_50' : 24,
            'roi_at_100' : 2,
            'roi_at_150' : 18,
            'roi_at_200' : 15,
            'roi_at_300' : 13,
            'roi_at_500' : 10}
        # Define SIGNAL_11 function
        def SIGNAL_11(df_ens):
            return df_ens['vcews']
        # Define MyCandlesStrat_11 class
        class MyCandlesStrat_11(Strategy):  
            sl_pct_l = best_params['BEST_STOP_LOSS_sl_pct_long']
            tp_pct_l = best_params['BEST_TAKE_PROFIT_tp_pct_long']
            limit_l = best_params['BEST_LIMIT_ORDER_limit_long']
            sl_pct_s = best_params['BEST_STOP_LOSS_sl_pct_short']
            tp_pct_s = best_params['BEST_TAKE_PROFIT_tp_pct_short']
            limit_s = best_params['BEST_LIMIT_ORDER_limit_short']
            margin_leverage = best_params['BEST_LEVERAGE_margin_leverage']
            TRAILING_ACTIVATE_PCT = best_params['TRAILING_ACTIVATE_PCT']
            TRAILING_STOP_PCT = best_params['TRAILING_STOP_PCT']
            roi_at_50 = best_params['roi_at_50']
            roi_at_100 = best_params['roi_at_100']
            roi_at_150 = best_params['roi_at_150']
            roi_at_200 = best_params['roi_at_200']
            roi_at_300 = best_params['roi_at_300']
            roi_at_500 = best_params['roi_at_500']
            def init(self):
                super().init()
                self.signal1 = self.I(SIGNAL_11, self.data)
                self.entry_time = Timestamp.now()
                self.max_profit = 0
            def next(self):
                super().next() 
                if (self.signal1 == 1):
                    sl_price = self.data.Close[-1] * (1 - (self.sl_pct_l * 0.001))
                    tp_price = self.data.Close[-1] * (1 + (self.tp_pct_l * 0.001))
                    limit_price_l = tp_price * 0.994
                    self.position.is_long
                    self.buy(sl=sl_price, limit=limit_price_l, tp=tp_price)
                    if self.position.is_long:
                        self.entry_time = self.trades[0].entry_time  # Accessing the current datetime
                    # Calculate current profit
                    # current_profit = self.trades[0].pl_pct
                    # Check for trailing stop loss based on current profit
                    if self.position and self.trades[0].pl_pct >= (self.TRAILING_ACTIVATE_PCT * 0.001):
                        self.max_profit = max(self.max_profit, self.trades[0].pl_pct)
                        trailing_stop_price = self.trades[0].entry_price * (1 + (self.max_profit - (self.TRAILING_STOP_PCT * 0.001)))
                        sl_price = min((self.data.Close[-1] * (1 - (self.TRAILING_STOP_PCT * 0.001))), trailing_stop_price)
                    # time_spent_by_asset1 = (self.data.index[-1] - self.trades[0].entry_time).total_seconds() / 60
                    # Check for time interval-based selling
                    if self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 50) and (self.trades[0].pl_pct > (self.roi_at_50 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 50) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 100) and (self.trades[0].pl_pct > (self.roi_at_100 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 100) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 150) and (self.trades[0].pl_pct > (self.roi_at_150 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 150) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 200) and (self.trades[0].pl_pct > (self.roi_at_200 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 200) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 300) and (self.trades[0].pl_pct > (self.roi_at_300 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 300) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166< 950) and (self.trades[0].pl_pct > (self.roi_at_500 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166>= 950):
                        self.position.close()
                elif (self.signal1 == 2):
                    sl_price = self.data.Close[-1] * (1 + (self.sl_pct_s * 0.001))
                    tp_price = self.data.Close[-1] * (1 - (self.tp_pct_s * 0.001))
                    limit_price_s = tp_price * 1.004
                    self.position.is_short
                    self.sell(sl=sl_price, limit=limit_price_s, tp=tp_price)
                    if self.position.is_short:
                        self.entry_time = self.trades[0].entry_time  # Accessing the current datetime
                    # Calculate current profit
                    # current_profit = self.trades[0].pl_pct
                    # Check for trailing stop loss based on current profit
                    if self.position and self.trades[0].pl_pct >= (self.TRAILING_ACTIVATE_PCT * 0.001):
                        self.max_profit = max(self.max_profit, self.trades[0].pl_pct)
                        trailing_stop_price = self.trades[0].entry_price * (1 - (self.max_profit - (self.TRAILING_STOP_PCT * 0.001)))
                        sl_price = max((self.data.Close[-1] * (1 - (self.TRAILING_STOP_PCT * 0.001))), trailing_stop_price)
                        time_spent_by_asset1 = (self.data.index[-1] - self.trades[0].entry_time).total_seconds() / 60
                    # Check for time interval-based selling
                    if self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 50) and (self.trades[0].pl_pct > (self.roi_at_50 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 50) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 100) and (self.trades[0].pl_pct > (self.roi_at_100 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 100) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 150) and (self.trades[0].pl_pct > (self.roi_at_150 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 150) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 200) and (self.trades[0].pl_pct > (self.roi_at_200 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 200) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166<= 300) and (self.trades[0].pl_pct > (self.roi_at_300 * 0.001)):
                        self.position.close()
                    elif self.position  and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166> 300) and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166< 950) and (self.trades[0].pl_pct > (self.roi_at_500 * 0.001)):
                        self.position.close()
                    elif self.position and ((self.data.index[-1] - self.trades[0].entry_time).total_seconds()  *  0.0166>= 950):
                        self.position.close()

        # Run backtest with optimized parameters
        bt_11 = Backtest(df_ens, MyCandlesStrat_11, cash=100000, commission=.001, margin=(1 / MyCandlesStrat_11.margin_leverage), exclusive_orders=False)
        stat_11 = bt_11.run()
        print("stat_11 line 388 - ", stat_11)
        # Additional processing for custom_assets
        # custom_assets = {}
        if ((stat_11['Return [%]'] > (stat_11['Buy & Hold Return [%]'] * 3)) 
            & (stat_11['Profit Factor'] > 1.0)
            & (stat_11['Max. Drawdown [%]'] > -35)
            & (stat_11['Win Rate [%]'] > 52)
            & (stat_11['Return [%]'] > 0)):
            file_prefix = file_path.split('/')[-1].split('.')[0]
            
            print(f"second backtest success for {file_prefix}/USDT:USDT with Win Rate % of {stat_11['Win Rate [%]']} and with Return in % of {stat_11['Return [%]']}" )
            
            
            best_params = {'Optimizer': '2nd backtest with Expectancy',
            # 'OptimizerResult_Cross': f"2nd backtest, Sharpe Ratio - {stat_11['Sharpe Ratio']}, Returns % - {stat_11['Return [%]']}, Win Rate % - {stat_11['Win Rate [%]']}",
                           'model_trained_on': model_name,
            'OptimizerResult_Cross': f"For {file_prefix}/USDT:USDT backtest was done from {stat_11['Start']} upto {stat_11['End']} for a duration of {stat_11['Duration']} using time frame of {timeframe} with Win Rate % - {round(stat_11['Win Rate [%]'],2)}, Return % - {round(stat_11['Return [%]'],3)}, Expectancy % - {round(stat_11['Expectancy [%]'],5)} and Sharpe Ratio - {round(stat_11['Sharpe Ratio'],3)}.",
            'BEST_STOP_LOSS_sl_pct_long': MyCandlesStrat_11.sl_pct_l.tolist(),
            'BEST_TAKE_PROFIT_tp_pct_long': MyCandlesStrat_11.tp_pct_l.tolist(),
            'BEST_LIMIT_ORDER_limit_long':  round(MyCandlesStrat_11.tp_pct_l.tolist() * 0.996, 2),
            'BEST_STOP_LOSS_sl_pct_short': MyCandlesStrat_11.sl_pct_s.tolist(),
            'BEST_TAKE_PROFIT_tp_pct_short': MyCandlesStrat_11.tp_pct_s.tolist(),
            'BEST_LIMIT_ORDER_limit_short': round(MyCandlesStrat_11.sl_pct_s.tolist() * 0.996,2),
            'BEST_LEVERAGE_margin_leverage': MyCandlesStrat_11.margin_leverage.tolist(),
            'TRAILING_ACTIVATE_PCT': MyCandlesStrat_11.TRAILING_ACTIVATE_PCT.tolist(),
            'TRAILING_STOP_PCT' : MyCandlesStrat_11.TRAILING_STOP_PCT.tolist(),
            'roi_at_50' : MyCandlesStrat_11.roi_at_50.tolist(),
            'roi_at_100' : MyCandlesStrat_11.roi_at_100.tolist(),
            'roi_at_150' :MyCandlesStrat_11.roi_at_150.tolist(),
            'roi_at_200' : MyCandlesStrat_11.roi_at_200.tolist(),
            'roi_at_300' : MyCandlesStrat_11.roi_at_300.tolist(),
            'roi_at_500' : MyCandlesStrat_11.roi_at_500.tolist()
                          }
            
            # print("best_params under stat_11 ", best_params)
            key_mapping = {
                'Optimizer': 'Optimizer_used',
                'model_trained_on': 'model_name',
                'OptimizerResult_Cross': 'Optimizer_result',
                'BEST_STOP_LOSS_sl_pct_long': 'stop_loss_percent_long',
                'BEST_TAKE_PROFIT_tp_pct_long': 'take_profit_percent_long',
                'BEST_LIMIT_ORDER_limit_long': 'limit_long',
                'BEST_STOP_LOSS_sl_pct_short': 'stop_loss_percent_short',
                'BEST_TAKE_PROFIT_tp_pct_short': 'take_profit_percent_short',
                'BEST_LIMIT_ORDER_limit_short': 'limit_short',
                'BEST_LEVERAGE_margin_leverage': 'margin_leverage',
                'TRAILING_ACTIVATE_PCT': 'TRAILING_ACTIVATE_PCT',
                'TRAILING_STOP_PCT' : 'TRAILING_STOP_PCT',
                'roi_at_50' : 'roi_at_50',
                'roi_at_100' : 'roi_at_100',
                'roi_at_150' :'roi_at_150',
                'roi_at_200' : 'roi_at_200',
                'roi_at_300' : 'roi_at_300',
                'roi_at_500' : 'roi_at_500'
            }
            # Update or add new entry to custom_assets
            custom_assets = load_custom_assets()
            
            transformed_params = {}
            for old_key, value in best_params.items():
                new_key = key_mapping.get(old_key, old_key)
                transformed_params[new_key] = value
            new_key = file_prefix + "/USDT:USDT"
            # custom_assets[new_key] = transformed_params
            if new_key in custom_assets:
                # Update existing entry
                for key, value in transformed_params.items():
                    if isinstance(value, (int, float)) and key != 'margin_leverage' and value >= 1:
                        transformed_params[key] = round(transformed_params[key] * 0.001, 5)
                custom_assets[new_key].update(transformed_params)
            else:
                # Add new entry
                # Multiply numerical values by 0.001 for new entry if value > 1
                for key, value in transformed_params.items():
                    if isinstance(value, (int, float)) and key != 'margin_leverage' and value >= 1:
                        transformed_params[key] = round(transformed_params[key] * 0.001, 5)
                custom_assets[new_key] = transformed_params
            # Save custom_assets to JSON file
            save_custom_assets(custom_assets)
        print("custom_assets after save ", custom_assets)
    return df, symbol_name, custom_assets
    # except Exception as e:
    #     # Print the error message
    #     print(f"Error processing {file_path}: {e}")
    #     print("custom assets at error level line 361 ", custom_assets)
    #     # Return None for both DataFrame and symbol name to indicate failure
    #     return None, symbol_name, custom_assets

# Define a thread worker function
def thread_worker(file):
    result = process_json(file)
    return result
def main():
    # Get a list of all JSON files in the folder
    # NOTE: make sure to mention the tradingview downloaded data folder here
    json_files = [f"./tradingview_crypto_assets_15m/{file}" for file in os.listdir("./tradingview_crypto_assets_15m/") if file.endswith(".json")]
    # print(json_files)
    # Get the number of available CPU cores
    num_cores = os.cpu_count()
    # print(num_cores)
    # Set the max_workers parameter based on the number of CPU cores
    max_workers = (num_cores) if (num_cores > 1) else 1  # Default to 1 if CPU count cannot be determined
    # max_workers = 1  # Default to 1 if CPU count cannot be determined
    print('max workers (Total Number of CPU cores to be used) - ', max_workers)
    # Process JSON files in parallel using multi-core processing
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Submit threads for each JSON file
        futures = [executor.submit(thread_worker, file) for file in json_files]
    # Wait for all threads to complete
    results = [future.result() for future in futures]
    # Process the results as needed
    for result in results:
        if result is None:
            continue
        df, symbol_name, custom_assets = result
        print(f"Processed {symbol_name}")
        print(f'custom_assets ', custom_assets)
        if custom_data:  # Check if custom_data is not None
            custom_assets.update(custom_data)
            
# Define a function to continuously run the loop
def run_continuous_loop():
    while True:
        main()
# Start the continuous loop in a separate thread
thread = threading.Thread(target=run_continuous_loop)
thread.start()

output:
max workers (Total Number of CPU cores to be used) -  4
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step
219/219 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step
backtest one done at 226 line -  Start                     2024-03-02 11:45:00
End                       2024-05-14 03:45:00
Duration                     72 days 16:00:00
Exposure Time [%]                   85.237208
Equity Final [$]                  45917.74697
Equity Peak [$]                  119511.93047
Return [%]                         -54.082253
Buy & Hold Return [%]              -27.134777
Return (Ann.) [%]                  -98.222272
Volatility (Ann.) [%]                3.390676
Sharpe Ratio                              0.0
Sortino Ratio                             0.0
Calmar Ratio                              0.0
Max. Drawdown [%]                  -63.780594
Avg. Drawdown [%]                   -7.944307
Max. Drawdown Duration       65 days 12:15:00
Avg. Drawdown Duration        6 days 13:06:00
# Trades                                  704
Win Rate [%]                        42.471591
Best Trade [%]                       7.078622
Worst Trade [%]                     -5.342172
Avg. Trade [%]                      -0.100692
Max. Trade Duration           0 days 16:00:00
Avg. Trade Duration           0 days 02:09:00
Profit Factor                        0.910244
Expectancy [%]                      -0.083294
SQN                                 -1.448338
_strategy                    MyCandlesStrat_3
_equity_curve                             ...
_trades                          Size  Ent...
dtype: object

backtest one done at 226 line -  Start                     2024-03-02 11:45:00
End                       2024-05-14 03:45:00
Duration                     72 days 16:00:00
Exposure Time [%]                   68.295829
Equity Final [$]                  78570.91076
Equity Peak [$]                 102494.351527
Return [%]                         -21.429089
Buy & Hold Return [%]               10.347826
Return (Ann.) [%]                  -70.858652
Volatility (Ann.) [%]               51.978975
Sharpe Ratio                              0.0
Sortino Ratio                             0.0
Calmar Ratio                              0.0
Max. Drawdown [%]                  -41.288206
Avg. Drawdown [%]                  -15.066559
Max. Drawdown Duration       71 days 23:15:00
Avg. Drawdown Duration       24 days 03:40:00
# Trades                                  219
Win Rate [%]                        57.990868
Best Trade [%]                      17.226306
Worst Trade [%]                     -5.348487
Avg. Trade [%]                        0.01822
Max. Trade Duration           2 days 10:30:00
Avg. Trade Duration           0 days 09:08:00
Profit Factor                        1.061187
Expectancy [%]                        0.12345
SQN                                 -0.480221
_strategy                    MyCandlesStrat_3
_equity_curve                             ...
_trades                          Size  Ent...
dtype: object
backtest one done at 226 line -  Start                     2024-03-02 11:45:00
End                       2024-05-14 03:45:00
Duration                     72 days 16:00:00
Exposure Time [%]                   77.210836
Equity Final [$]                255299.429466
Equity Peak [$]                 263013.301626
Return [%]                         155.299429
Buy & Hold Return [%]              -27.134777
Return (Ann.) [%]                13205.285493
Volatility (Ann.) [%]            18553.318471
Sharpe Ratio                         0.711748
Sortino Ratio                      269.760311
Calmar Ratio                       559.102219
Max. Drawdown [%]                  -23.618732
Avg. Drawdown [%]                    -2.60826
Max. Drawdown Duration       21 days 15:00:00
Avg. Drawdown Duration        0 days 17:31:00
# Trades                                  285
Win Rate [%]                         68.77193
Best Trade [%]                       7.078622
Worst Trade [%]                     -5.355119
Avg. Trade [%]                       0.427783
Max. Trade Duration           2 days 05:30:00
Avg. Trade Duration           0 days 08:32:00
Profit Factor                        1.408437
Expectancy [%]                       0.486645
SQN                                  1.913964
_strategy                    MyCandlesStrat_3
_equity_curve                             ...
_trades                           Size  En...
dtype: object
Error decoding JSON in custom_assets.txt: Expecting value: line 1 column 1 (char 0)
{'NKN/USDT:USDT': {'Optimizer_used': '1st backtest - Expectancy', 'model_name': 'model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras', 'Optimizer_result': 'For NKN/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 68.77, Return % - 155.299,Expectancy % - 0.48665 and Sharpe Ratio - 0.7117.', 'stop_loss_percent_long': 0.052, 'take_profit_percent_long': 0.055, 'limit_long': 0.054, 'stop_loss_percent_short': 0.052, 'take_profit_percent_short': 0.055, 'limit_short': 0.054, 'margin_leverage': 1, 'TRAILING_ACTIVATE_PCT': 0.045, 'TRAILING_STOP_PCT': 0.005, 'roi_at_50': 0.054, 'roi_at_100': 0.05, 'roi_at_150': 0.045, 'roi_at_200': 0.04, 'roi_at_300': 0.03, 'roi_at_500': 0.01}}
backtest one done at 226 line -  Start                     2024-03-02 11:45:00
End                       2024-05-14 03:45:00
Duration                     72 days 16:00:00
Exposure Time [%]                   74.874588
Equity Final [$]                128738.734226
Equity Peak [$]                 159762.295359
Return [%]                          28.738734
Buy & Hold Return [%]               -7.367375
Return (Ann.) [%]                  225.143219
Volatility (Ann.) [%]              319.059091
Sharpe Ratio                         0.705647
Sortino Ratio                        4.823065
Calmar Ratio                        10.486484
Max. Drawdown [%]                  -21.469848
Avg. Drawdown [%]                   -4.048638
Max. Drawdown Duration       13 days 09:15:00
Avg. Drawdown Duration        1 days 18:33:00
# Trades                                  183
Win Rate [%]                        68.306011
Best Trade [%]                       6.186671
Worst Trade [%]                     -5.332174
Avg. Trade [%]                       0.294714
Max. Trade Duration           2 days 18:30:00
Avg. Trade Duration           0 days 11:01:00
Profit Factor                        1.262946
Expectancy [%]                       0.362138
SQN                                  0.587677
_strategy                    MyCandlesStrat_3
_equity_curve                             ...
_trades                        Size  Entry...
dtype: object
{'NKN/USDT:USDT': {'Optimizer_used': '1st backtest - Expectancy', 'model_name': 'model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras', 'Optimizer_result': 'For NKN/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 68.77, Return % - 155.299,Expectancy % - 0.48665 and Sharpe Ratio - 0.7117.', 'stop_loss_percent_long': 0.052, 'take_profit_percent_long': 0.055, 'limit_long': 0.054, 'stop_loss_percent_short': 0.052, 'take_profit_percent_short': 0.055, 'limit_short': 0.054, 'margin_leverage': 1, 'TRAILING_ACTIVATE_PCT': 0.045, 'TRAILING_STOP_PCT': 0.005, 'roi_at_50': 0.054, 'roi_at_100': 0.05, 'roi_at_150': 0.045, 'roi_at_200': 0.04, 'roi_at_300': 0.03, 'roi_at_500': 0.01}, 'NEO/USDT:USDT': {'Optimizer_used': '1st backtest - Expectancy', 'model_name': 'model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras', 'Optimizer_result': 'For NEO/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 68.31, Return % - 28.739,Expectancy % - 0.36214 and Sharpe Ratio - 0.7056.', 'stop_loss_percent_long': 0.052, 'take_profit_percent_long': 0.055, 'limit_long': 0.054, 'stop_loss_percent_short': 0.052, 'take_profit_percent_short': 0.055, 'limit_short': 0.054, 'margin_leverage': 1, 'TRAILING_ACTIVATE_PCT': 0.045, 'TRAILING_STOP_PCT': 0.005, 'roi_at_50': 0.054, 'roi_at_100': 0.05, 'roi_at_150': 0.045, 'roi_at_200': 0.04, 'roi_at_300': 0.03, 'roi_at_500': 0.01}}



.................................................................................................................................
(output goes on for all the assets and then short listed assets get saved inside custom_assets.txt)

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

The provided Python code appears to be related to backtesting a cryptocurrency trading strategy. Here’s a breakdown of the code functionalities:

Data Processing:

1. Function process_json: This function reads a JSON file containing cryptocurrency price data.
2. Data Cleaning and Transformation: It cleans and transforms the data by:

• Renaming columns to standard names (e.g., ‘date’ to ‘Date’).
• Converting the ‘Date’ column to datetime format.
• Setting ‘Date’ as the index.
• Filling missing values in the ‘Close’ column with the previous close price.
• Extracting the symbol name from the ‘symbol’ column.

1. Technical Indicator Calculation: The script calculates various technical indicators like ATR, EMA, RSI, etc., using the ta library (assumed to be imported).
2. Feature Engineering: It creates additional features like returns, volatility, volume-based indicators, and momentum-based indicators.
3. Data Scaling: The script scales the data using MinMaxScaler for better model performance during backtesting.
4. Reshaping Data: The data is reshaped into a format suitable for the trading strategy (e.g., sequences of past price data).

Backtesting Strategy:

1. Function SIGNAL_3: This function likely defines the trading signals based on some criteria (not shown in the provided code).
2. Class MyCandlesStrat_3: This class defines the trading strategy using the Backtrader library (assumed to be imported). Key elements include:

• Stop-loss and Take-profit: These are set based on predefined percentages (BEST_STOP_LOSS_sl_pct_long, etc.) for long and short positions.
• Limit orders: These are used to ensure order execution within a specific price range.
• Trailing Stop-loss: The stop-loss is dynamically adjusted based on current profit to lock in gains.
• Time-based profit taking: Profits are automatically locked in after a certain time holding the asset.
• Leverage: The strategy uses a predefined leverage multiplier (BEST_LEVERAGE_margin_leverage).

Backtesting and Analysis:

1. Backtest: The script performs a backtest on the processed data using the MyCandlesStrat_3 strategy with a starting capital of 100000.
2. Performance Metrics: Backtesting results likely include various performance metrics like returns, Sharpe Ratio, Win Rate, and Drawdown (not explicitly shown in the provided code).

Conditional Logic:

• The script checks if certain performance conditions are met (high return, good profit factor, etc.).
• If the conditions are satisfied, the script potentially saves the trading strategy parameters for this specific asset.

Usage of ThreadPoolExecutor class for parallel processing of JSON files. Here's a breakdown of its functionality:

1. Thread Worker Function (thread_worker):

• This function takes a single JSON file path as input (file).
• It calls the process_json function (assumed to be defined elsewhere) to process the JSON data.
• It returns the processed result, likely a Pandas DataFrame (df), symbol name (symbol_name), and potentially other custom data (custom_assets).

2. Main Function (main):

• It retrieves a list of all JSON files within a specified folder (./tradingview_crypto_assets_15m/).
• It determines the number of available CPU cores using os.cpu_count().
• It sets the max_workers parameter for the ThreadPoolExecutor based on the CPU cores (using all cores if available, defaulting to 1 otherwise).
• It prints the number of cores to be used for processing.
• It creates a ThreadPoolExecutor with the determined max_workers.
• It iterates through the list of JSON files and submits each file path to the thread pool using executor.submit(thread_worker, file). This creates tasks for each file to be processed concurrently.
• It waits for all submitted tasks (futures) to complete using future.result() and stores the results in a list (results).
• It iterates through the processing results:
• If a result is None, it skips to the next iteration (potentially handling errors).
• Otherwise, it unpacks the result (df, symbol_name, and potentially custom_assets).
• It prints information about the processed symbol and the custom assets (if any).
• It conditionally updates custom_assets with additional custom data (custom_data) if it exists (logic not entirely shown).

3. Continuous Loop Function (run_continuous_loop):

• This function defines an infinite loop (while True).
• Inside the loop, it calls the main function, presumably to process a batch of JSON files repeatedly.

4. Starting the Loop:

• The code creates a separate thread using threading.Thread and sets its target to the run_continuous_loop function.
• Finally, it starts the thread, initiating the continuous processing loop.

Overall, this code snippet demonstrates parallel processing of JSON files using a thread pool based on CPU cores. The loop continuously processes batches of files.

The code demonstrates a framework for backtesting a cryptocurrency trading strategy that uses technical indicators and incorporates risk management techniques like stop-loss and trailing stop-loss.

Disclaimer:

• Always remeber that, Backtesting results may not be indicative of future performance.
• Trading cryptocurrencies involves significant risks, and you should always do your own research before making any investment decisions.

custom_assets.txt Output:

{
    "NKN/USDT:USDT": {
        "Optimizer_used": "1st backtest - Expectancy",
        "model_name": "model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras",
        "Optimizer_result": "For NKN/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 68.77, Return % - 155.299,Expectancy % - 0.48665 and Sharpe Ratio - 0.7117.",
        "stop_loss_percent_long": 0.052,
        "take_profit_percent_long": 0.055,
        "limit_long": 0.054,
        "stop_loss_percent_short": 0.052,
        "take_profit_percent_short": 0.055,
        "limit_short": 0.054,
        "margin_leverage": 1,
        "TRAILING_ACTIVATE_PCT": 0.045,
        "TRAILING_STOP_PCT": 0.005,
        "roi_at_50": 0.054,
        "roi_at_100": 0.05,
        "roi_at_150": 0.045,
        "roi_at_200": 0.04,
        "roi_at_300": 0.03,
        "roi_at_500": 0.01
    },
    "NEO/USDT:USDT": {
        "Optimizer_used": "1st backtest - Expectancy",
        "model_name": "model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras",
        "Optimizer_result": "For NEO/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 68.31, Return % - 28.739,Expectancy % - 0.36214 and Sharpe Ratio - 0.7056.",
        "stop_loss_percent_long": 0.052,
        "take_profit_percent_long": 0.055,
        "limit_long": 0.054,
        "stop_loss_percent_short": 0.052,
        "take_profit_percent_short": 0.055,
        "limit_short": 0.054,
        "margin_leverage": 1,
        "TRAILING_ACTIVATE_PCT": 0.045,
        "TRAILING_STOP_PCT": 0.005,
        "roi_at_50": 0.054,
        "roi_at_100": 0.05,
        "roi_at_150": 0.045,
        "roi_at_200": 0.04,
        "roi_at_300": 0.03,
        "roi_at_500": 0.01
    },
    "AUDIO/USDT:USDT": {
        "Optimizer_used": "1st backtest - Expectancy",
        "model_name": "model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras",
        "Optimizer_result": "For AUDIO/USDT:USDT backtest was done from 2024-03-02 11:45:00 upto 2024-05-14 03:45:00 for a duration of 72 days 16:00:00 using time frame of 15m with Win Rate % - 67.05, Return % - 87.616,Expectancy % - 0.66847 and Sharpe Ratio - 0.5269.",
        "stop_loss_percent_long": 0.052,
        "take_profit_percent_long": 0.055,
        "limit_long": 0.054,
        "stop_loss_percent_short": 0.052,
        "take_profit_percent_short": 0.055,
        "limit_short": 0.054,
        "margin_leverage": 1,
        "TRAILING_ACTIVATE_PCT": 0.045,
        "TRAILING_STOP_PCT": 0.005,
        "roi_at_50": 0.054,
        "roi_at_100": 0.05,
        "roi_at_150": 0.045,
        "roi_at_200": 0.04,
        "roi_at_300": 0.03,
        "roi_at_500": 0.01
    },
    "ENA/USDT:USDT": {
        "Optimizer_used": "1st backtest - Expectancy",
        "model_name": "model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras",
        "Optimizer_result": "For ENA/USDT:USDT backtest was done from 2024-04-04 00:15:00 upto 2024-05-14 04:00:00 for a duration of 40 days 03:45:00 using time frame of 15m with Win Rate % - 70.0, Return % - 6.486,Expectancy % - 0.67719 and Sharpe Ratio - 0.3738.",
        "stop_loss_percent_long": 0.052,
        "take_profit_percent_long": 0.055,
        "limit_long": 0.054,
        "stop_loss_percent_short": 0.052,
        "take_profit_percent_short": 0.055,
        "limit_short": 0.054,
        "margin_leverage": 1,
        "TRAILING_ACTIVATE_PCT": 0.045,
        "TRAILING_STOP_PCT": 0.005,
        "roi_at_50": 0.054,
        "roi_at_100": 0.05,
        "roi_at_150": 0.045,
        "roi_at_200": 0.04,
        "roi_at_300": 0.03,
        "roi_at_500": 0.01
    },
    }
.................................... 
(all 30+assets got short listed as per paramter given by us during 
optimization and backtesting with downlaoded data for the neural 
network model we trained our model on)

The provided data snippet appears to be the results of backtesting a cryptocurrency trading strategy on multiple assets. Here’s a breakdown of the information:

Structure:

• It’s a dictionary with currency pairs (e.g., “MATIC/USDT:USDT”) as keys.

Content for Each Asset:

• Optimizer_used: This specifies the optimization method used for backtesting (here, “1st backtest — Expectancy”).
• model_name: This indicates the model name used for the trading strategy (“model_lstm_Balanced__15m_ETH_SL55_TP55_ShRa_0.71_time_20240529033537.keras”).
• Optimizer_result: This is a detailed description of the backtesting results for the specific asset. It includes:
• Start and end date of the backtest.
• Backtesting duration.
• Timeframe used (e.g., 15m).
• Win Rate percentage.
• Return percentage.
• Expectancy percentage.
• Sharpe Ratio.
• stop_loss_percent_long/short: These define the stop-loss percentages for long and short positions.
• take_profit_percent_long/short: These define the take-profit percentages for long and short positions.
• limit_long/short: These define the maximum price deviation allowed for entry orders (likely to prevent excessive slippage).
• margin_leverage: This specifies the leverage used for margin trading (set to 1 here, indicating no leverage).
• TRAILING_ACTIVATE_PCT & TRAILING_STOP_PCT: These define parameters for trailing stop-loss, which adjusts the stop-loss dynamically.
• roi_at_50, 100, 150, etc.: These are potentially profit targets at different holding durations (e.g., roi_at_50 might be the target profit for holding 50% of the time).

Interpretation:

• This data likely comes from a backtesting tool that evaluated a specific trading strategy on various cryptocurrencies.
• The results show performance metrics like win rate, return, and Sharpe Ratio for each asset.
• Stop-loss, take-profit, and leverage parameters define the risk management aspects of the strategy.

Shortlisted Assets and Saving:

• The statement mentions “shortlisted assets” but doesn’t explicitly show how they are identified. It’s possible that assets meeting certain performance criteria (based on the backtesting results) are considered shortlisted.
• These shortlisted assets are potentially saved in a file named “saved_assets.txt” in the same format as the provided data snippet.

Disclaimer:

• Backtesting results are not a guarantee of future performance.
• Trading cryptocurrencies involves significant risks, and you should always do your own research before making any investment decisions.

Conclusion:

The LSTM time series classification model has proven to be a valuable tool for predicting Ethereum price movements and generating profitable trading signals. Despite the high volatility and substantial drawdowns, the model’s robust returns and positive expectancy demonstrate its potential in algorithmic trading.

However, there are areas for further improvement and optimization:

1. Risk Management: Implementing advanced risk management techniques could help mitigate drawdowns and volatility.
2. Model Optimization: Continuous refinement of the LSTM model’s architecture and hyperparameters can enhance performance.
3. Broader Application: Extending the model to other assets and timeframes could provide additional insights and opportunities.
4. Live Trading: Testing the strategy in a live trading environment would provide practical insights and validate its real-world applicability.

Overall, the LSTM model’s ability to capture complex patterns in time series data makes it a powerful tool for trading strategies, with significant potential for generating high returns.

This article describes a cryptocurrency trading system that utilizes a neural network model (specifically a LSTM model) and a trading bot called VishvaAlgo.

Here’s a breakdown:

Data and Model Training:

• The system downloads historical data for over 250+ cryptocurrency assets on Binance Futures from TradingView.
• It trains a LSTM-based neural network model, achieving a claimed return of 700%+ on Ethereum (ETHUSDT) in 3 years on a 15-minutes time frame data with over 100,000 rows trained model with 193+ features used for finding the best possible estimation for going neutral, long and short using the classification based neural network LSTM model. (important to note: this returns vary from system to system based on trained data and needs re-verification).

Hyperparameter Optimization and Asset Selection:

• The system uses Hyperopt (a hyperparameter optimization library) to identify the most suitable assets for the trained model among the downloaded data.
• Each shortlisted asset has a unique set of parameters like stop-loss, take-profit, leverage, tailored for the model’s predictions.

VishvaAlgo — The Trading Bot:

• VishvaAlgo helps automate live trading using the trained model and the shortlisted assets with their pre-defined parameters.
• The bot offers easy integration with various neural network models for classification.
• A video explaining VishvaAlgo’s features and benefits is available — Link

Benefits of VishvaAlgo:

• Automates trading based on the trained model and optimized asset selection.
• Offers easy integration with user-defined neural network models.
• Provided detailed explanation and installation guide for purchase through my Patreon page.

Youtube Link Explanation of VishvaAlgo v4.x Features — Link

get entire code and profitable algos @ https://patreon.com/pppicasso

Disclaimer: Trading involves risk. Past performance is not indicative of future results. VishvaAlgo is a tool to assist traders and does not guarantee profits. Please trade responsibly and conduct thorough research before making investment decisions.

Warm Regards,

Puranam Pradeep Picasso

Linkedin — https://www.linkedin.com/in/puranampradeeppicasso/

Patreon — https://patreon.com/pppicasso

Facebook — https://www.facebook.com/puranam.p.picasso/

Twitter — https://twitter.com/picasso_999